Question

跑步时

>   spark-submit   --class "TwitterPopularTags"   --master local[2]      >/home/raja/begin/target/scala-2.11/simple-project_2.11-1.0.jar

>  "Exception in thread "main" java.lang.NoClassDefFoundError:                      >org/apache/spark/Logging
>...
>Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
>...
>"

我的sbt很好：干净，重新加载，编译，打包但是当我运行上面的spark-submit时，我得到了上面这个世界闻名的错误。

>My simple.sbt:

>    name := "Simple Project"
>    version := "1.0"
>    scalaVersion := "2.11.6"

>libraryDependencies ++= Seq(
>  "org.apache.spark" %% "spark-core" % "2.0.1",
>  "org.apache.spark" %% "spark-streaming" % "2.0.1",
>  "org.apache.spark" %% "spark-streaming-twitter" % "1.6.2",
>  "com.google.code.gson" % "gson" % "2.7",
>  "org.twitter4j" % "twitter4j-core" % "4.0.4",
>  "org.twitter4j" % "twitter4j-stream" % "4.0.4",
>"org.apache.logging.log4j" % "log4j-slf4j-impl" % "2.7"
>)

>resolvers += "Maven Central" at "https://repo1.maven.org/maven2/"
>resolvers += "Akka Repository" at "http://repo.akka.io/releases/"

我的进口商品：

>import org.apache.spark.streaming.StreamingContext._
>import org.apache.spark.streaming.dstream.DStream
>import org.apache.spark.streaming.twitter._
>import org.apache.spark.streaming.{Seconds, StreamingContext}
>import org.apache.spark.{SparkConf, SparkContext}

>import org.slf4j.{Logger, LoggerFactory}
>import org.slf4j.impl.StaticLoggerBinder
>import org.apache.log4j.{Level, LogManager, PropertyConfigurator}
>import org.apache.log4j.Logger
>import twitter4j.auth.OAuthAuthorization
>import twitter4j.conf.ConfigurationBuilder
>import twitter4j.Twitter
>import twitter4j.Status
>import twitter4j.auth.Authorization
>import twitter4j.TwitterFactory

请不要告诉我更改版本。我配置了log4j.properties，属性但没有机会。是SLF4J还是Logback我要看？我试图从maven获得该版本的spark-core jar，但没有机会。

任何人都可以指出我在这里

谢谢，拉加

Answer 1

您的依赖项不正确：

Spark 2.0
Spark Twitter连接器1.6

在Spark 2.0中删除了Twitter连接器。您可以使用Apache Bahir或将Spark降级为1.6。

我错过了任何进口吗？

1 个答案: