我在shell中设置以下导出。
export SPARK_HOME="/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/spark"
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/build:$PYTHONPATH
PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/build:$PYTHONPATH
确认SPARK_HOME文件夹内容：
ls /opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/spark -l
drwxr-xr-x 3 root root 4096 Aug 12 14:47 assembly
drwxr-xr-x 2 root root 4096 Aug 12 14:47 bin
drwxr-xr-x 2 root root 4096 Aug 12 14:47 cloudera
lrwxrwxrwx 1 root root 15 Dec 1 19:56 conf -> /etc/spark/conf
drwxr-xr-x 3 root root 4096 Aug 12 14:47 examples
drwxr-xr-x 2 root root 4096 Aug 12 14:52 lib
-rw-r--r-- 1 root root 46083 Aug 12 14:47 LICENSE
-rw-r--r-- 1 root root 22559 Aug 12 14:47 NOTICE
drwxr-xr-x 7 root root 4096 Aug 12 14:47 python
-rw-r--r-- 1 root root 0 Aug 12 14:47 RELEASE
drwxr-xr-x 2 root root 4096 Aug 12 14:47 sbin
drwxr-xr-x 3 root root 4096 Aug 12 14:47 ui-resources
lrwxrwxrwx 1 root root 19 Dec 1 19:56 work -> /var/run/spark/work
将以下代码作为 spark-submit TestPyEnv.py
运行import os
import sys
# Path for spark source folder
#os.environ['SPARK_HOME']="/opt/cloudera/parcels/CDH/lib/spark"
# Append pyspark to Python Path
#sys.path.append("/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/lib/spark/python/")
help('modules')
try:
from pyspark import SparkContext
from pyspark import SparkConf
print ("Successfully imported Spark Modules")
except ImportError as e:
print ("Can not import Spark Modules", e)
sys.exit(1)
获得以下输出：
Please wait a moment while I gather a list of all available modules...
/usr/lib64/python2.6/site-packages/matplotlib/numerix/__init__.py:18: DeprecationWarning:
**********************************************************
matplotlib.numerix and all its subpackages are deprecated.
They will be removed soon. Please use numpy instead.
**********************************************************
warnings.warn(msg, DeprecationWarning)
** (TestPyEnv.py:13441): WARNING **: Trying to register gtype 'WnckWindowState' as flags when in fact it is of type 'GEnum'
** (TestPyEnv.py:13441): WARNING **: Trying to register gtype 'WnckWindowActions' as flags when in fact it is of type 'GEnum'
** (TestPyEnv.py:13441): WARNING **: Trying to register gtype 'WnckWindowMoveResizeMask' as flags when in fact it is of type 'GEnum'
2015-12-11 17:58:50.887115: ERROR: Could not load the stocks from /home/irmprot/.gnome2/invest-applet/stocks.pickle: [Errno 2] No such file or directory: '/home/irmprot/.gnome2/invest-applet/stocks.pickle'
BaseHTTPServer calendar ipalib repr
Bastion cas ipapython resource
CDROM cgi itertools rexec
CGIHTTPServer cgitb iwlib rfc822
CORBA chunk ixf86config rhn
ConfigParser cmath json rhsm
Cookie cmd kerberos rlcompleter
Crypto code keyword robotparser
DLFCN codecs krbV rpm
DocXMLRPCServer codeop ldap rpmUtils
HTMLParser collections ldapurl runpy
IN colorsys ldif scdate
IRM_RAW_inputPreprocessor commands lib2to3 sched
LoadBaseAndIntegrated compileall libproxy select
M2Crypto compiler libsvn selinux
MimeWriter contextlib libuser sets
ORBit cookielib libxml2 setuptools
OpenSSL copy libxml2mod sgmllib
PortableServer copy_reg linecache sha
Pyspark-Step1-Dec9 cracklib linuxaudiodev shelve
Pyspark-Step1-latest crypt locale shlex
Queue csv logging shutil
SSSDConfig ctypes lxml signal
SimpleHTTPServer curl macpath simplejson
SimpleXMLRPCServer curses macurl2path site
SocketServer datetime mailbox slip
StringIO dateutil mailcap smtpd
TYPES dbhash markupbase smtplib
TestPyEnv dbm marshal snack
UserDict dbus math sndhdr
UserList dbus_bindings matplotlib socket
UserString decimal md5 sos
_LWPCookieJar default_encoding_utf8 meh spwd
_MozillaCookieJar difflib mhlib sqlite3
__builtin__ dircache mimetools sqlitecachec
__future__ dis mimetypes sre
_abcoll distutils mimify sre_compile
_ast dl mmap sre_constants
_bisect dmidecode modulefinder sre_parse
_bsddb dmidecodemod mpl_toolkits ssl
_bytesio doctest multifile stat
_codecs drv_libxml2 multiprocessing statvfs
_codecs_cn dsextras mutex string
_codecs_hk dsml netaddr stringold
_codecs_iso2022 dumbdbm netrc stringprep
_codecs_jp dummy_thread new strop
_codecs_kr dummy_threading nis struct
_codecs_tw easy_install nntplib subprocess
_collections egg nose sunau
_cracklib email nss sunaudio
_crypt encodings ntpath svn
_csv errno nturl2path symbol
_ctypes ethtool numbers symtable
_curses exceptions numpy sys
_curses_panel fcntl opcode syslog
_dbus_bindings filecmp operator system_config_keyboard
_dbus_glib_bindings fileinput optparse tabnanny
_elementtree firstboot orca talloc
_fileio fnmatch os tarfile
_functools formatter os2emxpath telnetlib
_hashlib fpformat ossaudiodev tempfile
_heapq fractions packagekit termios
_hotshot ftplib pango test
_json functools pangocairo tests
_ldap future_builtins paramiko textwrap
_locale gc parser this
_lsprof gconf pdb thread
_multibytecodec gdbm pickle threading
_multiprocessing genericpath pickletools time
_random getopt pipes timeit
_snack getpass pkg_resources timing
_socket gettext pkgutil toaiff
_sqlite3 gio platform token
_sqlitecache glib plistlib tokenize
_sre glob popen2 trace
_ssl gmenu poplib traceback
_strptime gnome posix tty
_struct gnomeapplet posixfile types
_symtable gnomecanvas posixpath unicodedata
_threading_local gnomevfs pprint unittest
_warnings gobject profile urlgrabber
_weakref gpgme pstats urllib
abc grp pty urllib2
abrt_exception_handler gtk pwd urlparse
acutil gtkunixprint py4j user
aifc gudev py_compile uu
anydbm gzip pyatspi uuid
array hashlib pyclbr vte
ast heapq pycurl warnings
asynchat hmac pydoc wave
asyncore hotshot pydoc_topics weakref
atexit htmlentitydefs pyexpat webbrowser
atk htmllib pygtk whichdb
audiodev httplib pyhbac wnck
audioop idlelib pylab wsgiref
base64 ihooks pyspark xdrlib
bdb imageop pysss xf86config
binascii imaplib pysss_murmur xml
binhex imghdr pytz xmllib
bisect imp quopri xmlrpclib
bonobo imputil random xxsubtype
bsddb iniparse re yum
bz2 inspect readline yumutils
cPickle invest redhat_support_lib zipfile
cProfile io redhat_support_tool zipimport
cStringIO iotop report zlib
cairo ipaclient reportclient
Enter any module name to get more help. Or, type "modules spam" to search
for modules whose descriptions contain the word "spam".
('Can not import Spark Modules', ImportError('cannot import name SparkContext',))
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/avro-tools-1.7.6-cdh5.4.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.4.5-1.cdh5.4.5.p0.7/jars/avro-tools-1.7.6-cdh5.4.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
我无法弄明白为什么SparkContext无法正常工作。
（＆＃39;无法导入Spark Modules＆＃39;，ImportError（＆＃39;无法导入名称SparkContext＆＃39;，））