我正在运行一个交互式pySpark会话,并且正在通知成千上万的此类消息
12: message written to socket.
90: took mesage out of queue: {"DATA":["application_1530518053043_0011","5","2","wn6-mtspar.ec1yjr0ydu0ebmgq53xysfleyg.fx.internal.cloudapp.net","2018-07-02T10:12:27.515+0000","",""],"SOURCE":"SparkExecutorEvents","TAG":"spark-metrics-d65d4762-8dc9-4c9d-a65f-a6af134c168c"}
90: message written to socket.
这使我无法理解代码的进度。
我尝试使用spark.sparkContext.setLogLevel('WARN')
,但是消息不会停止出现在屏幕上。