为什么spark-submit包引用始终不起作用?

时间:2015-12-01 00:27:00

标签: jar apache-spark hbase spark-streaming

所以,我遇到了许多人遇到的问题 - 请参阅下面的错误。

WARN scheduler.TaskSetManager: Lost task 0.0 in stage 3.0 (): java.io.IOException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace

有关修复此问题的说明似乎缺失,incompleteoverly complex。我只是通过将它们添加到spark-submit中的包列表中解决了其他参考问题。我也尝试过这种htrace依赖,但没有运气。

这是我目前的spark-submit命令:

spark-submit --class com.biz.HbasePassthrough \
        --packages \
            org.apache.spark:spark-streaming-kafka_2.10:1.3.0,org.apache.hbase:hbase-common:1.0.0,org.apache.hbase:hbase-client:1.0.0,org.apache.hbase:hbase-server:1.0.0,org.json4s:json4s-jackson_2.10:3.2.11,org.apache.htrace:htrace:3.1.0-incubating,org.apache.htrace:htrace-core:3.1.0-incubating,org.apache.htrace:htrace-hbase:3.1.0-incubating,org.apache.hbase:hbase-annotations:1.0.0 \
        ./my-spark_2.10-1.0.8.jar \
        >spark_log 2>&1

grep -n1 "htrace" spark_log

的输出
8-org.json4s#json4s-jackson_2.10 added as a dependency
9:org.apache.htrace#htrace added as a dependency
10:org.apache.htrace#htrace-core added as a dependency
11:org.apache.htrace#htrace-hbase added as a dependency
12-org.apache.hbase#hbase-annotations added as a dependency
--
36-     found org.mortbay.jetty#jetty-util;6.1.26 in central
37:     found org.apache.htrace#htrace-core;3.1.0-incubating in central
38-     found org.apache.hbase#hbase-client;1.0.0 in central
--
75-     found com.fasterxml.jackson.core#jackson-core;2.3.1 in central
76:     found org.apache.htrace#htrace;3.1.0-incubating in central
77:     found org.apache.htrace#htrace-hbase;3.1.0-incubating in central
78-:: resolution report :: resolve 6423ms :: artifacts dl 45ms
--
111-    org.apache.hbase#hbase-server;1.0.0 from central in [default]
112:    org.apache.htrace#htrace;3.1.0-incubating from central in [default]
113:    org.apache.htrace#htrace-core;3.1.0-incubating from central in [default]
114:    org.apache.htrace#htrace-hbase;3.1.0-incubating from central in [default]
115-    org.apache.kafka#kafka_2.10;0.8.1.1 from central in [default]
--
182-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/json4s-jackson_2.10.jar at http://192.168.240.209:37644/jars/json4s-jackson_2.10.jar with timestamp 1448926760838
183:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-core.jar at http://192.168.240.209:37644/jars/htrace-core.jar with timestamp 1448926760845
184:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-hbase.jar at http://192.168.240.209:37644/jars/htrace-hbase.jar with timestamp 1448926760846
185-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/hbase-annotations.jar at http://192.168.240.209:37644/jars/hbase-annotations.jar with timestamp 1448926760847

作业开始时的所有输出似乎表明它找到了包并将成功使用它,但它不起作用。

我终于找到了一些我认为让我超越问题的东西 - here。我通过在上面的spark-submit命令中添加以下两行来改编它

    --driver-class-path "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
    --conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \

为什么包参考不起作用?

0 个答案:

没有答案