Question

有没有办法禁用Spark streaming中显示的默认输出。下面给出了运行spark streaming Application时打印的默认输出。我想看到只有我添加的日志消息（在驱动程序中）才会被打印

spark-submit --class a11_TCP_IP.a11_Basic.a11_HelloTcp target/scala-2.10/sparkstreamingexamples_2.10-1.0.jar

16/04/27 13:20:15 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:15 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
16/04/27 13:20:17 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:17 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777618000 ms
-------------------------------------------

16/04/27 13:20:19 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data
16/04/27 13:20:19 ERROR ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Socket data stream had no more data
-------------------------------------------
Time: 1461777621000 ms
-------------------------------------------

16/04/27 13:20:21 WARN ReceiverSupervisorImpl: Restarting receiver with delay 2000 ms: Socket data stream had no more data

Answer 1

您的Spark文件夹应该有一个conf/目录。在其中，有一个log4j.properties文件。如果没有，请将conf/log4j.properties.template复制到conf/log4j.properties。

现在在编辑器中打开conf/log4j.properties，并替换此行：

log4j.rootCategory=INFO, console

这两个：

log4j.rootCategory=INFO, null
log4j.appender.null=org.apache.log4j.varia.NullAppender

Answer 2

我正在使用IntelliJ进行Spark Streaming，无法看到conf文件夹。因此，使用代码来调整日志记录。

Logger.getLogger（“ org”）。setLevel（Level.ERROR）//导入所需的库

停止Spark Streaming中的默认输出

2 个答案: