如下所示,Spark控制台输出进度条正在弄乱输出。是否有可用于关闭舞台进度条的配置或标志?或者更好的是,如何修复控制台日志,以便在完成阶段后进度条消失?这可能只是PySpark的一个错误,但我不确定。
(CID, (v1 / n1, v2 / n2))
[Stage 46:============================================> (19 + 4) / 24]('1', (0.020000000000000035, 4.805))
('5', (6.301249999999998, 0.125))
('10', (21.78000000000001, 3.125))
('7', (0.005000000000000009, 0.6049999999999996))
(CID, sqrt(v1 / n1 + v2 / n2))
('1', 2.19658826364888)
('5', 2.5350049309616733)
('10', 4.990490957811667)
('7', 0.7810249675906652)
(CID, (AD_MEAN, NCI_MEAN))
('7', (1.0, 5.5))
('5', (7.75, 5.3))
('10', (13.5, 6.0))
('1', (3.0, 5.0))
(CID, (AD_MEAN - NCI_MEAN))
('7', -4.5)
('5', 2.45)
('1', -2.0)
('10', 7.5)
(CID, (NUMER, DENOM))
[Stage 100:===================================================> (30 + 2) / 32]('10', (7.5, 4.990490957811667))
('5', (2.45, 2.5350049309616733))
('7', (-4.5, 0.7810249675906652))
('1', (-2.0, 2.19658826364888))
有时甚至更糟(滚动到右边):
$ spark-submit main.py
17/04/28 11:36:23 WARN Utils: Your hostname, Pandora resolves to a loopback address: 127.0.1.1; using 146.95.36.193 instead (on interface wlp3s0)
17/04/28 11:36:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
17/04/28 11:36:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[Stage 0:> (0 + 2 [Stage 32:=============================> (4 + 4[Stage 37:> (0 + 0[Stage 35:=====> (4 + 2) / 12][Stage 37:> (0 + 0[Stage 35:===========> (8 + 4) / 12][Stage 37:> (0 + 0[Stage 37:=======> (1 + 3[Stage 37:=============================> (4 + 0[Stage 36:========> (13 + 4) / 24][Stage 37:=========> (4 + 0[Stage 36:==============> (21 + 3) / 24][Stage 37:=========> (4 + 1[Stage 37:====================================> (5 + 3[Stage 38:===================================> (20 + 4)[Stage 38:====================================================> (30 + 2) SORTED (t-value, CID)
[(-5.761659596980321, '7'), (-0.9105029072119708, '1'), (0.9664675480810896, '5'), (1.5028581483070664, '10')]
答案 0 :(得分:3)
您可以通过设置
来禁用spark.ui.showConsoleProgress
= False 或
log4j.properties
的日志记录级别降至INFO
以上,即ERROR
相关的Spark jiras:
从版本1.2开始, spark.ui.showConsoleProgress
始终在Spark中,但仅在Spark 2.2中记录。
答案 1 :(得分:2)
Tagar的答案在pyspark中对我没有用。
以下是我发现从控制台中删除进度条的解决方法:
from pyspark import SparkContext, SparkConf
from pyspark.sql.session import SparkSession
conf = SparkConf().set("spark.ui.showConsoleProgress", "false")
sc = SparkContext(appName="RandomForest", conf=conf)
spark = SparkSession(sc)
希望这有帮助!
答案 2 :(得分:0)
以下是您使用pyspark 2.4.x
spark = SparkSession
.builder.master('local') \
.appName('MySparkApplication') \
.config('spark.ui.showConsoleProgress', 'false') \ # <=====
.getOrCreate()