Question

我正在使用带有spark依赖项的scala应用程序。在这里我有什么

log4j.properties

# Here we have defined root logger
log4j.rootLogger=WARN,ERROR,R

#Direct log messages to file
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=./logging.log
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p  %c{1}:%L- %m%n

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

# Silence akka remoting
#log4j.logger.Remoting=WARN

# Ignore messages below warning level from Jetty, because it's a bit verbose
#log4j.logger.org.eclipse.jetty=WARN

主要班级

org.apache.log4j.PropertyConfigurator.configure("./log4j.properties")

val sparkSession = SparkSession.builder.master("local").appName("spark session example").getOrCreate
val spark = sparkSession.sqlContext

Logger.getLogger("org").setLevel(Level.OFF)
Logger.getLogger("akka").setLevel(Level.OFF)

val log = Logger.getLogger(this.getClass.getName)
log.error("This is error");
log.warn("This is warn");
println("log ok")

的pom.xml

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
      <modelVersion>4.0.0</modelVersion>

      <groupId>com.ayd</groupId>
      <artifactId>data2</artifactId>
      <version>0.0.1-SNAPSHOT</version>
      <packaging>jar</packaging>

      <name>data2</name>
      <url>http://maven.apache.org</url>

      <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
      </properties>

      <dependencies>
       <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-catalyst_2.11</artifactId>
                <version>2.0.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-core_2.11</artifactId>
                <version>2.0.0</version>
            </dependency>

            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-sql_2.11</artifactId>
                <version>2.0.0</version>

            </dependency>

            <dependency>
                <groupId>org.postgresql</groupId>
                <artifactId>postgresql</artifactId>
                <version>42.1.3</version>
            </dependency>





            <!-- https://mvnrepository.com/artifact/log4j/log4j -->
            <dependency>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
                <version>1.2.17</version>
            </dependency>
        <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>3.8.1</version>
          <scope>test</scope>
        </dependency>
      </dependencies>
    </project>

我只是得到我显示的信息，而不是火花日志。当我这样做时，我在日志文件中发现了这个额外的火花日志消息

 2017-10-18 11:58:38 WARN   NativeCodeLoader:62- Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

        2017-10-18 11:58:41 ERROR  Run$:23- This is error
        2017-10-18 11:58:41 WARN   Run$:24- This is warn

如何摆脱日志文件中的第一行。

提前致谢

Answer 1

您可以将spark类的日志记录级别设置为WARN，这样您就会看到此警告。

如果要过滤掉所有Spark警告（不是真的推荐），可以在log4j属性文件中将级别设置为ERROR：

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

另一个警告来自Hadoop代码（包org.apache.hadoop） - 所以要禁用那个（以及所有其他Hadoop警告），您可以指定该包的级别ERROR：< / p>

log4j.logger.org.apache.hadoop=ERROR

Spark：显示日志消息

1 个答案: