Question

我正在尝试从hive上的表中运行一个简单的选择计数（*）。但是，在状态更改为STARTED后，我收到错误。

Hadoop版本 - 2.6.2 Spark版本 - 1.6.2 Hive版本 - 2.0.1

以下是完整的流程

hive> select count(*) from customers;
Query ID = hadoop_20170302215442_6493f265-3121-4eaf-b2d6-eea5b08ae591
Total jobs = 1
Launching Job 1 out of 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Spark Job = b22ec8e6-4b11-49fe-9fdd-001aa7248084
state = SENT
state = SENT
state = SENT
state = SENT
state = SENT
state = SENT
state = SENT
state = STARTED
state = STARTED
state = STARTED

Query Hive on Spark job[0] stages:
0
1

Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-    FailedTasksCount)/TotalTasksCount [StageCost]
2017-03-02 21:55:02,526 Stage-0_0: 0(+1)/1      Stage-1_0: 0/1
state = STARTED
2017-03-02 21:55:03,537 Stage-0_0: 0(+1,-1)/1   Stage-1_0: 0/1
state = FAILED
Status: Failed
FAILED: Execution Error, return code 3 from     org.apache.hadoop.hive.ql.exec.spark.SparkTask

在Yarn日志文件中，我看到以下错误..

Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, slcag034.us.oracle.com): java.lang.NoClassDefFoundError: Lorg/apache/hive/spark/counter/SparkCounters;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)

我在hive-site.xml中设置了spark.driver.extraClassPath

<property>
        <name>spark.driver.extraClassPath</name>
        <value>file:///***/***/binary/apache-hive-2.0.1-bin/lib/hive-exec-2.0.1.jar</value>
    </property>

有人可以告诉我可能出了什么问题。

此致

巴拉

hive执行引擎 - Spark - NoClassDefFoundError - SparkCounters

0 个答案: