Question

我用过火花1.2.1 hadoop 2.3.0-cdh5.0.2 hbase 0.96.1.1-cdh5.0.2

当我运行spark app时，它总是显示以下异常。其实我有 org.apache.hbase HBase的协议而protobuf-java 2.5.0已经构建为spark spark jar。

spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, cfswps1d-phys.nam.nsroot.net): java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/protobuf/generated/MasterProtos$MasterService$BlockingInterface
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:186)
        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:377)
        at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:366)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:247)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:188)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:150)
        at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:101)
        at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:130)
        at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:107)
        at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:69)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
        at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:61)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:245)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
        at org.apache.spark.scheduler.Task.run(Task.scala:56)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingInterface
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        ... 23 more

Answer 1

您需要将错误中提到的程序中引用的类$MasterService$BlockingInterface添加到Spark Conf文件中：

spark.driver.extraClassPath=

您可以在spark site上阅读有关extraClassPath选项的更多信息。

注意：在客户端模式下，不能直接在应用程序中通过SparkConf设置此配置，因为驱动程序JVM已在此时启动。相反，请通过--driver-class-path命令行选项或在默认属性文件中设置它。=“任何jar文件目录或类文件”

Spark中的Java HiveContext问题

1 个答案: