所以,我遇到了许多人遇到的问题 - 请参阅下面的错误。
WARN scheduler.TaskSetManager: Lost task 0.0 in stage 3.0 (): java.io.IOException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
有关修复此问题的说明似乎缺失,incomplete或overly complex。我只是通过将它们添加到spark-submit中的包列表中解决了其他参考问题。我也尝试过这种htrace依赖,但没有运气。
这是我目前的spark-submit命令:
spark-submit --class com.biz.HbasePassthrough \
--packages \
org.apache.spark:spark-streaming-kafka_2.10:1.3.0,org.apache.hbase:hbase-common:1.0.0,org.apache.hbase:hbase-client:1.0.0,org.apache.hbase:hbase-server:1.0.0,org.json4s:json4s-jackson_2.10:3.2.11,org.apache.htrace:htrace:3.1.0-incubating,org.apache.htrace:htrace-core:3.1.0-incubating,org.apache.htrace:htrace-hbase:3.1.0-incubating,org.apache.hbase:hbase-annotations:1.0.0 \
./my-spark_2.10-1.0.8.jar \
>spark_log 2>&1
grep -n1 "htrace" spark_log
8-org.json4s#json4s-jackson_2.10 added as a dependency
9:org.apache.htrace#htrace added as a dependency
10:org.apache.htrace#htrace-core added as a dependency
11:org.apache.htrace#htrace-hbase added as a dependency
12-org.apache.hbase#hbase-annotations added as a dependency
--
36- found org.mortbay.jetty#jetty-util;6.1.26 in central
37: found org.apache.htrace#htrace-core;3.1.0-incubating in central
38- found org.apache.hbase#hbase-client;1.0.0 in central
--
75- found com.fasterxml.jackson.core#jackson-core;2.3.1 in central
76: found org.apache.htrace#htrace;3.1.0-incubating in central
77: found org.apache.htrace#htrace-hbase;3.1.0-incubating in central
78-:: resolution report :: resolve 6423ms :: artifacts dl 45ms
--
111- org.apache.hbase#hbase-server;1.0.0 from central in [default]
112: org.apache.htrace#htrace;3.1.0-incubating from central in [default]
113: org.apache.htrace#htrace-core;3.1.0-incubating from central in [default]
114: org.apache.htrace#htrace-hbase;3.1.0-incubating from central in [default]
115- org.apache.kafka#kafka_2.10;0.8.1.1 from central in [default]
--
182-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/json4s-jackson_2.10.jar at http://192.168.240.209:37644/jars/json4s-jackson_2.10.jar with timestamp 1448926760838
183:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-core.jar at http://192.168.240.209:37644/jars/htrace-core.jar with timestamp 1448926760845
184:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-hbase.jar at http://192.168.240.209:37644/jars/htrace-hbase.jar with timestamp 1448926760846
185-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/hbase-annotations.jar at http://192.168.240.209:37644/jars/hbase-annotations.jar with timestamp 1448926760847
作业开始时的所有输出似乎表明它找到了包并将成功使用它,但它不起作用。
我终于找到了一些我认为让我超越问题的东西 - here。我通过在上面的spark-submit命令中添加以下两行来改编它
--driver-class-path "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
为什么包参考不起作用?