Question

我已经使用图像1.2在GCP中创建了Dataproc集群。我想从Datalab笔记本运行Spark。如果我保持Datalab笔记本运行Python 2.7作为其内核，则此方法很好，但是如果我想使用Python 3，则遇到次要版本不匹配的情况。我将在下面演示与Datalab脚本的不匹配：

### Configuration
import sys, os
sys.path.insert(0, '/opt/panera/lib')
os.environ['PYSPARK_PYTHON'] = '/opt/conda/bin/python'
os.environ['PYSPARK_DRIVER_PYTHON'] = '/opt/conda/bin/python'

import google.datalab.storage as storage
from io import BytesIO

spark = SparkSession.builder \
  .enableHiveSupport() \
  .config("hive.exec.dynamic.partition","true") \
  .config("hive.exec.dynamic.partition.mode","nonstrict") \
  .config("mapreduce.fileoutputcommitter.marksuccessfuljobs","false") \
  .getOrCreate() \

sc = spark.sparkContext

### import libraries
from pyspark.mllib.tree import DecisionTree, DecisionTreeModel
from pyspark.mllib.util import MLUtils
from pyspark.mllib.regression import LabeledPoint

### trivial example
data = [ 
  LabeledPoint(0.0, [0.0]),
  LabeledPoint(1.0, [1.0]),
  LabeledPoint(1.0, [2.0]),
  LabeledPoint(1.0, [3.0])
]

toyModel = DecisionTree.trainClassifier(sc.parallelize(data), 2, {})
print(toyModel)

错误：

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, pan-bdaas-prod-jrl6-w-3.c.big-data-prod.internal, executor 6): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/worker.py", line 124, in main
    ("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 3.6 than that in driver 3.5, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

其他初始化脚本： gs：//dataproc-initialization-actions/cloud-sql-proxy/cloud-sql-proxy.sh gs：//dataproc-initialization-actions/datalab/datalab.sh ...以及加载我们一些必要的库和实用程序的脚本

Answer 1

Datalab中的Python 3内核使用的是Python 3.5，而不是Python 3.6

您可以尝试在Datalab中设置一个3.6环境，然后为其安装新的kernelspec，但是仅配置Dataproc集群以使用Python 3.5可能会更容易

设置群集以使用3.5的说明为here

如何强制python版本在从GCP dataproc集群旋转的datalab实例中进行同步？

1 个答案: