Pyspark - 工人和工作室中不同的Python版本司机

Question

我在python3环境中使用pyspark。我有一个数据框，我正在尝试将一列密集的vectos拆分为多个列值。 我的df是这样的：

df_vector = kmeansModel_2.transform(finalData).select(['scalaredFeatures', 
                                                       'prediction'])
df_vector.show()

+--------------------+----------+
|    scalaredFeatures|prediction|
+--------------------+----------+
|[0.56785108466505...|         0|
|[1.41962771166263...|         0|
|[2.20042295307707...|         0|
|[0.14196277116626...|         0|
|[1.41962771166263...|         0|
+-------------------------------+

好吧，为了完成我的任务，我使用以下代码：

def extract(row):
    return (row.prediction, ) + tuple(row.scalaredFeatures.toArray().tolist())

df = df_vector.rdd.map(extract)toDF(["prediction"])

不幸的是我收到了错误：

Py4JJavaError: An error occurred while calling 
z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: 
Task 0 in stage 52.0 failed 1 times, most recent failure: Lost task 
0.0 in stage 52.0 (TID 434, localhost, executor driver): 
org.apache.spark.api.python.PythonException: Traceback (most recent 
call last):
 File "pyspark/worker.py", line 123, in main
("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 2.7 than that in 
driver 3.6, PySpark cannot run with different minor versions.Please 
check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON 
are correctly set.

有没有人可以帮我完成这项任务？谢谢！

Answer 1

如果使用PyCharm，则可以添加PYSPARK_PYTHON和PYSPARK_DRIVER_PYTHON来运行/调试配置。