Question

我正在尝试使用客户端模式在mesos主服务器上提交Spark作业。

我正在使用https://spark.apache.org/docs/latest/quick-start.html

中的示例

"""SimpleApp.py"""
from pyspark.sql import SparkSession

logFile = "YOUR_SPARK_HOME/README.md"  # Should be some file on your system
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
logData = spark.read.text(logFile).cache()

numAs = logData.filter(logData.value.contains('a')).count()
numBs = logData.filter(logData.value.contains('b')).count()

print("Lines with a: %i, lines with b: %i" % (numAs, numBs))

spark.stop()

这是火花提交命令

spark-submit --master mesos://HOST:5055 --deploy-mode client /Users/<username>/Documents/SimpleApp.py

当我杀死以下跟踪的提交时，该提交挂起

^CTraceback (most recent call last):
  File "/Users/<username>/Documents/SimpleApp.py", line 6, in <module>
    spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/session.py", line 173, in getOrCreate
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 367, in getOrCreate
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 136, in __init__
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 198, in _do_init
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 306, in _initialize_context
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1523, in __call__
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 985, in send_command
  File "/Users/<username>/Documents/install/spark-2.4.2-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1152, in send_command
  File "/anaconda3/lib/python3.7/socket.py", line 589, in readinto
19/05/24 11:51:23 INFO DiskBlockManager: Shutdown hook called
    return self._sock.recv_into(b)
KeyboardInterrupt

当我提交给本地Mesos i.e. mesos://localhost:5050/

时，同样的方法

任何对此的帮助都会很棒。

mesos master上的火花提交挂起

0 个答案: