PYspark SparkContext错误“调用None.org.apache.spark.api.java.JavaSparkContext时发生错误。”

时间:2018-01-18 22:07:25

标签: macos apache-spark terminal pyspark

我知道此问题已经发布过,但我尝试实施解决方案,但没有一个对我有用。我为Jupyter Notebook安装了Spark 使用本教程:

https://medium.com/@GalarnykMichael/install-spark-on-mac-pyspark-
453f395f240b#.be80dcqat

在MAC上安装最新版本的Apache Spark

当我尝试在Jupyter中运行以下代码时

wordcounts = sc.textFile('words.txt')

我收到以下错误:

name 'sc' is not defined

当我尝试添加代码时:

from pyspark import SparkContext, SparkConf
sc =SparkContext()

收到以下错误:

An error occurred while calling 
None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.hadoop.util.StringUtils
at
org.apache.hadoop.security.SecurityUtil.
getAuthenticationMethod(SecurityUtil.java:611)

在bash中添加了路径:

export SPARK_PATH=~/spark-2.2.1-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

#For python 3, You have to add the line below or you will get an error
# export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

请帮我解决这个问题。

0 个答案:

没有答案