Question

有人可以帮我创建Spark上下文时遇到错误吗？我收到以下错误时无法创建

from pyspark import SparkConf,SparkContext

conf=SparkConf().setMaster("localhost")
sc=SparkContext(conf = conf)


ExceptionTraceback (most recent call last)
<ipython-input-7-3d97d6624879> in <module>()
----> 1 sc=SparkContext(conf = conf)

/home/cloudera/anaconda2/lib/python2.7/site-packages/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    113         """
    114         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 115         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    116         try:
    117             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

/home/cloudera/anaconda2/lib/python2.7/site-packages/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway, conf)
    296         with SparkContext._lock:
    297             if not SparkContext._gateway:
--> 298                 SparkContext._gateway = gateway or launch_gateway(conf)
    299                 SparkContext._jvm = SparkContext._gateway.jvm
    300 

/home/cloudera/anaconda2/lib/python2.7/site-packages/pyspark/java_gateway.pyc in launch_gateway(conf)
     92 
     93             if not os.path.isfile(conn_info_file):
---> 94                 raise Exception("Java gateway process exited before sending its port number")
     95 
     96             with open(conn_info_file, "rb") 

Exception: Java gateway process exited before sending its port number

Answer 1

当您的PySpark版本与您设置的Spark版本不匹配时，可能会发生这种情况。（由public override void RegisterArea(AreaRegistrationContext context) { context.MapRoute( null, "/BackOffice/Brand/Act/Manage", new { controller = "Brand", action = "Edit" } ); }变量决定）-我记得很早以前就是这个问题。

在您的终端上尝试

SPARK_HOME

然后

pip freeze|grep pyspark

确保版本匹配。如果是这样，请重新启动内核，然后重试。希望这会有所帮助！

无法在Jupyter Notebook上创建Spark上下文

1 个答案: