Question

我使用eclipse IDE使用yarn-client模式向群集提交远程spark任务。

但在eclipse IDE中收到以下错误消息。

java.lang.NoClassDefFoundError: com/thoughtworks/paranamer/BytecodeReadingParanamer
    com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.<init>(BeanIntrospector.scala:40)
    com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.<clinit>(BeanIntrospector.scala)
    com.fasterxml.jackson.module.scala.introspect.ScalaPropertiesCollector.<init>(ScalaPropertiesCollector.scala:22)
    com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospector$.constructPropertyCollector(ScalaClassIntrospector.scala:24)
    com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:142)
    com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:68)
    com.fasterxml.jackson.databind.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:11)
    com.fasterxml.jackson.databind.SerializationConfig.introspect(SerializationConfig.java:490)
    com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:133)
    com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:873)
    com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:833)
    com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:387)
    com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:478)
    com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:97)
    com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:2718)
    com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2210)
    org.apache.spark.rdd.RDDOperationScope.toJson(RDDOperationScope.scala:51)
    org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:144)
    org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    org.apache.spark.SparkContext.withScope(SparkContext.scala:714)
    org.apache.spark.SparkContext.textFile(SparkContext.scala:830)
    org.apache.spark.api.java.JavaSparkContext.textFile(JavaSparkContext.scala:181)
    com.nuevora.core.spark.commons.CommonFunctions.getJavaRDDFromFile(CommonFunctions.java:61)
    com.nuevora.core.spark.UpdateDataset.modifyInputDataset(UpdateDataset.java:103)
    com.nuevora.controllers.FormsValidatorServlet.service(FormsValidatorServlet.java:3070)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

执行程序日志没有帮助。但最终的工作状态是“成功”

需要帮助解决此错误

Answer 1

我通过添加带有cloudera分布的paranamer jar解决了这个问题

从eclipse IDE提交远程spark工作，得到paranamer错误

1 个答案: