Question

我在intelliJ上运行我的spark工作，并且收到以下错误： -

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/sanchay/brazil-pkg-cache/packages/Log4j-slf4j/Log4j-slf4j-2.x.698.1338/RHEL5_64/DEV.STD.PTHREAD/build/lib/log4j-slf4j-impl-2.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/sanchay/brazil-pkg-cache/packages/Slf4j_log4j/Slf4j_log4j-1.7.2463.17539/RHEL5_64/DEV.STD.PTHREAD/build/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/sanchay/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.

但是，当我使用spark-submit运行相同的jar / class时，我能够获得所需的所有INFO日志： -

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/11/08 00:17:27 INFO SparkContext: Running Spark version 2.2.0
17/11/08 00:17:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/11/08 00:17:28 INFO SparkContext: Submitted application: JavaSparkPi
17/11/08 00:17:28 INFO SecurityManager: Changing view acls to: sanchay
17/11/08 00:17:28 INFO SecurityManager: Changing modify acls to: sanchay
17/11/08 00:17:28 INFO SecurityManager: Changing view acls groups to: 
17/11/08 00:17:28 INFO SecurityManager: Changing modify acls groups to: 
17/11/08 00:17:28 INFO SecurityManager: SecurityManager: authentication dis

如何切换到IntelliJ中的默认日志文件，以便通过INFO日志获取有关Spark Ui等的信息？

Answer 1

假设您使用maven管理代码。构建依赖项：树并检查使用的日志库版本。 https://www.slf4j.org/legacy.html图表应该是非常好的图表。所有其他人都应被排除在外
如何构建jar以提交给spark？我假设您没有包含所有依赖项，因此您没有额外的日志记录库。

在IntelliJ中启用Spark日志

1 个答案: