我正在尝试使用python和visual studio代码设置apache-spark。到目前为止,我一直按照教程进行操作,但是我的代码没有出现任何错误,但是当我尝试运行代码时
spark-submit rdd/testingCode.py
我得到了很多错误和警告。它还说
can't open file '/usr/local/opt/python-spark-tutorial/rdd/testingCode.py': [Errno 2] No such file or directory'
尽管此路径确实存在该文件!
我相信这可能与我的python路径有关?请帮助!
这是我要运行的代码:
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName("word count").setMaster("local[3]")
sc = SparkContext(conf = conf)
inputIntegers = list(range(1,6))
integerRdd = sc.parallelize(inputIntegers)
for x in integerRdd.collect():
print(x)
这是错误:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by
org.apache.hadoop.security.authentication.util.KerberosUtil
(file:/usr/local/opt/spark/spark-2.3.3-bin-hadoop2.7/jars/hadoop-
auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of
org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further
illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2019-05-22 12:17:13 WARN NativeCodeLoader:62 - Unable to load
native-hadoop library for your platform... using builtin-java
classes where applicable
/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python:
can't open file '/usr/local/opt/python-spark-
tutorial/rdd/testingCode.py': [Errno 2] No such file or directory
2019-05-22 12:17:13 INFO ShutdownHookManager:54 - Shutdown hook
called
2019-05-22 12:17:13 INFO ShutdownHookManager:54 - Deleting
directory
/private/var/folders/fn/mf51ynh97v71r4609l_l1j4m0000gn/T/spark-bdd580eb-468c-430e-bc27-8a2ca7d89697