Question

在纱线簇上提交作业时，纱线日志中未显示任何应用日志。一切都在本地模式下工作。

显示以下错误：

StatusLogger Log4j2 could not find a logging implementation

我提供了以下火花提交

：

--driver-library-path /opt/spark-extras/apache-log4j-2.11.2-bin \
    --conf spark.driver.extraLibraryPath=/opt/spark-extras/apache-log4j-2.11.2-bin \
    --conf spark.executor.extraLibraryPath=/opt/spark-extras/apache-log4j-2.11.2-bin \

spark-defaults.conf指定log4j2.xml：

spark.executor.extraJavaOptions ...  -Dlog4j.configurationFile=log4j2.xml

xml文件：

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
        </Console>
    </Appenders>

    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console" />
        </Root>
    </Loggers>
</Configuration>

然后我的yarn日志上仅显示以下内容：

2019-06-05 16:45:28,495 [main] WARN  org.apache.spark.SparkConf - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used anymore
2019-06-05 16:45:46,298 [Driver] WARN  org.apache.spark.SparkConf - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used anymore
2019-06-05 16:45:46,314 [Driver] WARN  org.apache.spark.SparkConf - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used anymore
2019-06-05 16:45:46,315 [Driver] WARN  org.apache.spark.SparkConf - The configuration key 'spark.executor.port' has been deprecated as of Spark 2.0.0 and may be removed in the future. Not used anymore
2019-06-05 16:45:48,324 [Driver] WARN  org.apache.spark.scheduler.FairSchedulableBuilder - Fair Scheduler configuration file not found so jobs will be scheduled in FIFO order. To use fair scheduling, configure pools in fairscheduler.xml or set spark.scheduler.allocation.file to a file that contains the configuration.
2019-06-05 16:45:49,977 [Reporter] INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for : 
2019-06-05 16:45:50,654 [ContainerLauncher-2] INFO  org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0
2019-06-05 16:45:53,692 [Reporter] INFO  org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl - Received new token for :

我如何配置该spark-submit作业以使其在本地模式下工作时使用log4j2进行记录？

在纱线簇上提交火花作业时，Log4j不显示日志

0 个答案: