Question

我目前正在尝试将数据从Hive加载到ElasticSearch。我正在使用cloudera CDH 5.3。我已经将hadoop-es hive 2.0.2 jar添加到我的蜂巢路径中。我在10.44.162.169上启动并运行ElasticSearch 1.4.4。

我现在有一个名为hive_cdr的表，其中包含以下属性：

 traffic_type_id (big int)
 appelant (int)
 called_number (int)
 call_duration (int)
 location_number (string)
 date_heure_appel(string)

我正在尝试在我的配置单元中定义ES表以加载一些数据。为此，我做到了这一点：

CREATE EXTERNAL TABLE es_hive_cdr (
traffic bigint ,
calling int ,
called int ,
duration int ,
location string ,
date string )
ROW FORMAT SERDE 'org.elasticsearch.hadoop.hive.EsSerDe'
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES (
'es.nodes'='10.44.162.169',
'es.resource'='indexCDR/typeCDR'
) ;

但是，我得到了这个例外，说EsStorage无法识别。

我删除了EsStorage行并执行以试图找出发生了什么。

现在尝试将我的hive_cdr表中的数据加载到我的新表中：

insert into table es_hive_cdr2
select
traffic_type_id,
appelant,
called_number,
call_duration,
location_number,
date_heure_appel
from hive_cdr;

但它失败了，我收到了这个错误：

处理语句时出错：FAILED：执行错误，从org.apache.hadoop.hive.ql.exec.mr.MapRedTask返回代码2

阶段依赖：

  Stage-1 is a root stage
  Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5
  Stage-4
  Stage-0 depends on stages: Stage-4, Stage-3, Stage-6
  Stage-2 depends on stages: Stage-0
  Stage-3
  Stage-5
  Stage-6 depends on stages: Stage-5

舞台计划：

  Stage: Stage-1
    Map Reduce
      Map Operator Tree:
          TableScan
            alias: hive_cdr
            Statistics: Num rows: 267130 Data size: 58768736 Basic stats: COMPLETE Column stats: NONE
            Select Operator
              expressions: traffic_type_id (type: bigint), appelant (type: int), called_number (type: int), call_duration (type: int), location_number (type: string), date_heure_appel (type: string)
              outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
              Statistics: Num rows: 267130 Data size: 58768736 Basic stats: COMPLETE Column stats: NONE
              File Output Operator
                compressed: false
                Statistics: Num rows: 267130 Data size: 58768736 Basic stats: COMPLETE Column stats: NONE
                table:
                    input format: org.apache.hadoop.mapred.TextInputFormat
                    output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    serde: org.elasticsearch.hadoop.hive.EsSerDe
                    name: default.es_hive_cdr2

  Stage: Stage-7
    Conditional Operator

  Stage: Stage-4
    Move Operator
      files:
          hdfs directory: true
          destination: hdfs://master:8020/user/hive/warehouse/es_hive_cdr2/.hive-staging_hive_2015-03-02_14-09-08_285_4734041865540737822-2/-ext-10000

  Stage: Stage-0
    Move Operator
      tables:
          replace: false
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              serde: org.elasticsearch.hadoop.hive.EsSerDe
              name: default.es_hive_cdr2

  Stage: Stage-2
    Stats-Aggr Operator

  Stage: Stage-3
    Map Reduce
      Map Operator Tree:
          TableScan
            File Output Operator
              compressed: false
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.elasticsearch.hadoop.hive.EsSerDe
                  name: default.es_hive_cdr2

  Stage: Stage-5
    Map Reduce
      Map Operator Tree:
          TableScan
            File Output Operator
              compressed: false
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  serde: org.elasticsearch.hadoop.hive.EsSerDe
                  name: default.es_hive_cdr2

  Stage: Stage-6
    Move Operator
      files:
          hdfs directory: true
          destination: hdfs://master:8020/user/hive/warehouse/es_hive_cdr2/.hive-staging_hive_2015-03-02_14-09-08_285_4734041865540737822-2/-ext-10000

我真的需要一些帮助和指导，并且能够感激并感谢你！

Answer 1

尝试提供表格属性。

TBLPROPERTIES（＆＃39; es.resource＆＃39; =＆＃39; myviews / myview＆＃39;，＆＃39; es.nodes＆＃39; =＆＃39; hostname-of-es-cluster＆＃ 39;，＆＃39; es.port＆＃39; =＆＃39; 9200＆＃39;，＆＃39; es.input.json＆＃39; =＆＃39; false＆＃39;，＆＃39; es .write.operation＆＃39; =＆＃39; index＆＃39;，＆＃39; es.index.auto.create＆＃39; =＆＃39; yes＆＃39;，＆＃39; es.nodes.wan .only＆＃39; =＆＃39; true＆＃39;）;

同时将elasticsearch.yml文件中的属性更改为低于一个

network.host：_site _

无法将数据从Hive加载到ElasticSearch

处理语句时出错：FAILED：执行错误，从org.apache.hadoop.hive.ql.exec.mr.MapRedTask返回代码2

1 个答案: