Question

当我在Cloudera Yarn环境中使用spark-submit命令时，我遇到了这种异常：

java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.getDeclaredMethods(Class.java:1975)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.com$fasterxml$jackson$module$scala$introspect$BeanIntrospector$$listMethods$1(BeanIntrospector.scala:93)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findMethod$1(BeanIntrospector.scala:99)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.com$fasterxml$jackson$module$scala$introspect$BeanIntrospector$$findGetter$1(BeanIntrospector.scala:124)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3$$anonfun$apply$5.apply(BeanIntrospector.scala:177)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3$$anonfun$apply$5.apply(BeanIntrospector.scala:173)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3.apply(BeanIntrospector.scala:173)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$$anonfun$3.apply(BeanIntrospector.scala:172)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
    at scala.collection.immutable.List.foreach(List.scala:318)
...

spark-submit命令如下：

spark-submit --master yarn-cluster \
        --num-executors $2 \
        --executor-cores $3 \
        --class "APP" \
        --deploy-mode cluster \
        --properties-file $1 \
        --files $HDFS_PATH/log4j.properties,$HDFS_PATH/metrics.properties \
        --conf spark.metrics.conf=metrics.properties \
        APP.jar

请注意，TopicAndPartition.class位于着色的APP.jar中。

Answer 1

请尝试使用 - jars 选项添加Kafka jar，如下例所示：

spark-submit --master yarn-cluster \
    --num-executors $2 \
    --executor-cores $3 \
    --class "APP" \
    --deploy-mode cluster \
    --properties-file $1 \
    --jars /path/to/kafka.jar
    --files $HDFS_PATH/log4j.properties,$HDFS_PATH/metrics.properties \
    --conf spark.metrics.conf=metrics.properties \
    APP.jar

Answer 2

使用某些方法后，事实证明问题是因为版本不兼容而引起的。正如@ user1050619所说，请确保kafka，spark，zookeeper和scala的版本相互兼容。

spark throws java.lang.NoClassDefFoundError：kafka / common / TopicAndPartition

2 个答案: