我知道此问题已经发布过,但我尝试实施解决方案,但没有一个对我有用。我为Jupyter Notebook安装了Spark 使用本教程:
https://medium.com/@GalarnykMichael/install-spark-on-mac-pyspark-
453f395f240b#.be80dcqat
在MAC上安装最新版本的Apache Spark
当我尝试在Jupyter中运行以下代码时
wordcounts = sc.textFile('words.txt')
我收到以下错误:
name 'sc' is not defined
当我尝试添加代码时:
from pyspark import SparkContext, SparkConf
sc =SparkContext()
收到以下错误:
An error occurred while calling
None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.util.StringUtils
at
org.apache.hadoop.security.SecurityUtil.
getAuthenticationMethod(SecurityUtil.java:611)
在bash中添加了路径:
export SPARK_PATH=~/spark-2.2.1-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
#For python 3, You have to add the line below or you will get an error
# export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'
请帮我解决这个问题。