Question

我是PySpark和EMR的新手我试图通过Jupyter笔记本访问在EMR集群上运行的Spark，但遇到错误。

我使用以下代码生成SparkSession：

spark = SparkSession.builder \
    .master("local[*]")\
    .appName("Carbon - SingleWell parallelization on Spark")\
    .getOrCreate()

尝试以下访问远程群集，但它出错了：

spark = SparkSession.builder \
    .master("spark://<remote-emr-ec2-hostname>:7077")\
    .appName("Carbon - SingleWell parallelization on Spark")\
    .getOrCreate()

错误：

Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NullPointerException
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:567)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

任何帮助解决此问题都将非常感激。

Answer 1

EMR clusters have Jupyter and JupyterHub provisioned for you since EMR version 5.14.0.

Most likely, it is easier to tune those provisioned services up with some extra bootstrap actions than to wire up your local process to talk to the EMR master node.

Jupyter + EMR + Spark - 从本地机器上的Jupyter笔记本连接到EMR集群

1 个答案: