Question

我使用cloudera Manager部署了一个集群，并安装了Spark Parcel，在shell中键入pyspark时，它可以工作，但是在jupyter上运行以下代码会引发异常

代码

import sys
import py4j
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
conf = SparkConf()
conf.setMaster('yarn-client')
conf.setAppName('SPARK APP')
sc = SparkContext(conf=conf)
# sc= SparkContext.getOrCreate()
# sc.stop()

def mod(x):
    import numpy as np
    return (x, np.mod(x, 2))

rdd = sc.parallelize(range(1000)).map(mod).take(10)
print (rdd)

例外

/usr/lib/python3.6/site-packages/pyspark/context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    187         self._accumulatorServer = accumulators._start_update_server(auth_token)
    188         (host, port) = self._accumulatorServer.server_address
--> 189         self._javaAccumulator = self._jvm.PythonAccumulatorV2(host, port, auth_token)
    190         self._jsc.sc().register(self._javaAccumulator)
    191 

TypeError: 'JavaPackage' object is not callable

Answer 1

经过一番搜索后，spark使用的版本1.6与python 3.7不兼容，必须使用python 2.7来运行它

如何在纱线客户端上从jupyter运行火花

1 个答案: