Question

我是Scala的初学者，我正在尝试在Scala中运行模型，但遇到了一些问题：

这是文件：

package com.salesforce.hw.titanic

import com.salesforce.op._
import com.salesforce.op.features.FeatureBuilder
import com.salesforce.op.features.types._
import com.salesforce.op.readers.DataReaders
import com.salesforce.op.stages.impl.classification._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, LogManager}



/**
 * A minimal Titanic Survival example with TransmogrifAI
 */



object OpTitanicMini {

  case class Passenger
  (
    id: Long,
    survived: Double,
    pClass: Option[Long],
    name: Option[String],
    sex: Option[String],
    age: Option[Double],
    sibSp: Option[Long],
    parCh: Option[Long],
    ticket: Option[String],
    fare: Option[Double],
    cabin: Option[String],
    embarked: Option[String]
  )



  def main(args: Array[String]): Unit = {
    LogManager.getLogger("com.salesforce.op").setLevel(Level.ERROR)
    implicit val spark = SparkSession.builder.config(new SparkConf()).getOrCreate()
    import spark.implicits._

    // Read Titanic data as a DataFrame
    val pathToData = Option(args(0))
    val passengersData = DataReaders.Simple.csvCase[Passenger](pathToData, key = _.id.toString).readDataset().toDF()

    // Automated feature engineering
    val (survived, features) = FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived")
    val featureVector = features.toSeq.autoTransform()

    // Automated feature selection
    val checkedFeatures = survived.sanityCheck(featureVector, checkSample = 1.0, removeBadFeatures = true)

    // Automated model selection
    val (pred, raw, prob) = BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput()
    val model = new OpWorkflow().setInputDataset(passengersData).setResultFeatures(pred).train()

    println("Model summary:\n" + model.summaryPretty())








  }

}

当我尝试运行它时，出现此错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/LogManager
  at com.salesforce.hw.titanic.OpTitanicMini$.main(OpTitanicMini.scala:72)
  at com.salesforce.hw.titanic.OpTitanicMini.main(OpTitanicMini.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.LogManager
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 2 more

我尝试查看此问题并找到了blog post，我尝试了该博文中所说的内容：

我的log4j.properties文件如下：

log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.Remoting=ERROR
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# Change this to set Hadoop log level
log4j.logger.org.apache.hadoop=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

# Breeze
log4j.logger.breeze.optimize=FATAL

# BLAS & LAPACK
log4j.logger.com.github.fommil.netlib=ERROR

# TransmogrifAI logging
log4j.logger.com.salesforce.op=INFO
log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=OFF

# Helloworld logging
log4j.logger.com.salesforce.hw=INFO

我尝试了博客文章中提到的步骤，但仍然面临着同样的问题，我该如何解决这个问题？

Answer 1

LogManager类带有Spark依赖项之一。确保在运行时对您的类路径具有org.apache.spark:spark-core，org.apache.spark:spark-mlib，org.apache.spark:spark-sql及其所有传递依赖。

我们有一个示例sbt项目here，您可以看一下。

Scala：线程“ main”中的异常java.lang.NoClassDefFoundError：org / apache / log4j / LogManager

1 个答案: