PySpark例外:Java网关进程在发送其端口号之前已退出

时间:2019-01-14 10:00:21

标签: python apache-spark pyspark

我运行Windows 10,并通过Anaconda3安装了Python3。我正在使用Jupyter Notebook。我已经安装了Spark from here(spark-2.3.0-bin-hadoop2.7.tgz)。我已提取文件并将其粘贴到目录D:\ Spark中。我已经修改了环境变量:

用户变量:

变量:SPARK_HOME

值:D:\ Spark

系统变量

变量:PATH

值:D:\ Spark \ bin

我已经通过conda安装/更新了以下模块:

熊猫

numpy

骨髓

pyspark

py4j

已安装Java:

enter image description here

我不知道这是否相关,但是在我的环境变量中出现以下两个变量:

enter image description here

完成所有这些操作后,我重新启动,然后运行以下代码,这会导致错误消息,并将其粘贴到此处:

import pandas as pd

import seaborn as sns

# These lines enable the run of spark commands

from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
sc = SparkContext('local')
spark = SparkSession(sc)

import pyspark

data = sns.load_dataset('iris')

data_sp = spark.createDataFrame(data)

data_sp.show()

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-1-ec964ecd39a2> in <module>()
      7 from pyspark.context import SparkContext
      8 from pyspark.sql.session import SparkSession
----> 9 sc = SparkContext('local')
     10 spark = SparkSession(sc)
     11 

C:\ProgramData\Anaconda3\lib\site-packages\pyspark\context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    113         """
    114         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 115         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    116         try:
    117             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

C:\ProgramData\Anaconda3\lib\site-packages\pyspark\context.py in _ensure_initialized(cls, instance, gateway, conf)
    296         with SparkContext._lock:
    297             if not SparkContext._gateway:
--> 298                 SparkContext._gateway = gateway or launch_gateway(conf)
    299                 SparkContext._jvm = SparkContext._gateway.jvm
    300 

C:\ProgramData\Anaconda3\lib\site-packages\pyspark\java_gateway.py in launch_gateway(conf)
     92 
     93             if not os.path.isfile(conn_info_file):
---> 94                 raise Exception("Java gateway process exited before sending its port number")
     95 
     96             with open(conn_info_file, "rb") as info:

Exception: Java gateway process exited before sending its port number

如何使PySpark工作?

1 个答案:

答案 0 :(得分:1)

我已按照以下说明解决了该问题:https://changhsinlee.com/install-pyspark-windows-jupyter/