如何修复apache-spark

时间:2019-05-22 20:05:18

标签: python apache-spark

我正在尝试使用python和visual studio代码设置apache-spark。到目前为止,我一直按照教程进行操作,但是我的代码没有出现任何错误,但是当我尝试运行代码时

spark-submit rdd/testingCode.py 

我得到了很多错误和警告。它还说

can't open file '/usr/local/opt/python-spark-tutorial/rdd/testingCode.py': [Errno 2] No such file or directory' 

尽管此路径确实存在该文件!

我相信这可能与我的python路径有关?请帮助!

这是我要运行的代码:

from pyspark import SparkContext, SparkConf

conf = SparkConf().setAppName("word count").setMaster("local[3]")
sc = SparkContext(conf = conf)

inputIntegers = list(range(1,6))
integerRdd = sc.parallelize(inputIntegers)
for x in integerRdd.collect():
    print(x)

这是错误:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by 
org.apache.hadoop.security.authentication.util.KerberosUtil 
(file:/usr/local/opt/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop- 
auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of 
org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further 
illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2019-05-22 12:17:13 WARN  NativeCodeLoader:62 - Unable to load 
native-hadoop library for your platform... using builtin-java 
classes where applicable

/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python:

can't open file '/usr/local/opt/python-spark- 
tutorial/rdd/testingCode.py': [Errno 2] No such file or directory
2019-05-22 12:17:13 INFO  ShutdownHookManager:54 - Shutdown hook 
called
2019-05-22 12:17:13 INFO  ShutdownHookManager:54 - Deleting 
directory 
/private/var/folders/fn/mf51ynh97v71r4609l_l1j4m0000gn/T/spark-bdd580eb-468c-430e-bc27-8a2ca7d89697

0 个答案:

没有答案