我在hive上运行查询....似乎是从日志,它失败,因为表[date_dim]中的错误记录导致转换异常....当我在[date_dim]中查找其他记录时,没有什么不同特定记录(最后参见样本记录)。
指向特定记录我可能是错的...试图理解为什么会出现这种投射错误以及如何解决这个问题?
你能帮我理解为什么这个特殊记录的错误?怎么解决? 任何帮助都非常感谢!!!
Hive查询:
SELECT 'store' AS channel, 'ss_cdemo_sk' AS col_name, d_year, d_qoy, i_category, ss_ext_sales_price AS ext_sales_price
FROM store_sales , item , date_dim
WHERE ss_cdemo_sk IS NULL
AND ss_sold_date_sk = d_date_sk
AND ss_item_sk = i_item_sk ;
Hive日志:
Query ID = root_20150922151717_a94d4679-224b-41f3-8336-a799d4ebedab
Total jobs = 4
....
Launching Job 3 out of 4
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1441794795162_13426, Tracking URL = http://myhost:8035/proxy/application_1441794795162_13426/
Kill Command = /opt/hes/hadoop/hadoop-2.6.0//bin/hadoop job -kill job_1441794795162_13426
Hadoop job information for Stage-6: number of mappers: 1; number of reducers: 0
2015-09-22 15:18:34,291 Stage-6 map = 0%, reduce = 0%
2015-09-22 15:18:54,541 Stage-6 map = 100%, reduce = 0%
Ended Job = job_1441794795162_13426 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1441794795162_13426_m_000000 (and more) from job job_1441794795162_13426
Task with the most failures(4):
Task ID:task_1441794795162_13426_m_000000
URL : myhost:8088/taskdetails.jsp?jobid=job_1441794795162_13426&tipid=task_1441794795162_13426_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row **{"d_date_sk":2452538,"d_date_id":"AAAAAAAAKDMGFCAA","d_date":"2002-09-20","d_month_seq":1232,"d_week_seq":5360,"d_quarter_seq":412,"d_year":2002,"d_dow":5,"d_moy":9,"d_dom":20,"d_qoy":3,"d_fy_year":2002,"d_fy_quarter_seq":412,"d_fy_week_seq":5360,"d_day_name":"Friday ","d_quarter_name":"N","d_holiday":"2002Q3","d_weekend":"N","d_following_holiday":"Y","d_first_dom":2452761,"d_last_dom":2452519,"d_same_day_ly":2452447,"d_same_day_lq":2452173,"d_current_day":"N","d_current_week":"N","d_current_month":"N","d_current_quarter":"N","d_current_year":"N"}**
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"d_date_sk":2452538,"d_date_id":"AAAAAAAAKDMGFCAA","d_date":"2002-09-20","d_month_seq":1232,"d_week_seq":5360,"d_quarter_seq":412,"d_year":2002,"d_dow":5,"d_moy":9,"d_dom":20,"d_qoy":3,"d_fy_year":2002,"d_fy_quarter_seq":412,"d_fy_week_seq":5360,"d_day_name":"Friday ","d_quarter_name":"N","d_holiday":"2002Q3","d_weekend":"N","d_following_holiday":"Y","d_first_dom":2452761,"d_last_dom":2452519,"d_same_day_ly":2452447,"d_same_day_lq":2452173,"d_current_day":"N","d_current_week":"N","d_current_month":"N","d_current_quarter":"N","d_current_year":"N"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception: org.apache.hadoop.hive.serde2.io.HiveDecimalWritable cannot be cast to org.apache.hadoop.io.IntWritable
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:311)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveDecimalWritable cannot be cast to org.apache.hadoop.io.IntWritable
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual.evaluate(GenericUDFOPEqual.java:84)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:185)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:86)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:68)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:185)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:106)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:638)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:651)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:654)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:750)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 15 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
ATTEMPT: Execute BackupTask: org.apache.hadoop.hive.ql.exec.mr.MapRedTask
.......
Stage-Stage-8: Map: 2 Cumulative CPU: 42.59 sec HDFS Read: 122983367 HDFS Write: 5267230 SUCCESS
Stage-Stage-6: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Stage-Stage-2: Map: 2 Reduce: 1 Cumulative CPU: 8.78 sec HDFS Read: 15648221 HDFS Write: 8106 SUCCESS
Total MapReduce CPU Time Spent: 51 seconds 370 msec
OK
channel col_name d_year d_qoy i_category ext_sales_price
store ss_cdemo_sk 1998 1 Sports NULL
.....
.
.
.
..
来自date_dim的表格样本数据(包括错误记录)
hive> select * from date_dim;
date_dim.d_date_sk date_dim.d_date_id date_dim.d_date date_dim.d_month_seq date_dim.d_week_seq date_dim.d_quarter_seq date_dim.d_year date_dim.d_dow date_dim.d_moy date_dim.d_dom date_dim.d_qoy date_dim.d_fy_year date_dim.d_fy_quarter_seq date_dim.d_fy_week_seq date_dim.d_day_name date_dim.d_quarter_name date_dim.d_holiday date_dim.d_weekend date_dim.d_following_holiday date_dim.d_first_dom date_dim.d_last_dom date_dim.d_same_day_ly date_dim.d_same_day_lq date_dim.d_current_day date_dim.d_current_week date_dim.d_current_month date_dim.d_current_quarter date_dim.d_current_year
2452538 AAAAAAAAKDMGFCAA 2002-09-20 1232 5360 412 2002 5 9 20 3 2002 412 5360 Friday N 2002Q3 N Y 2452761 2452519 2452447 2452173 N N N NN
2431208 AAAAAAAAIOIBFCAA 1944-04-27 531 2313 178 1944 4 4 27 2 1944 178 2313 Thursday N 1944Q2 N N 2431272 2431182 2431117 2430842 N N N NN
2456494 AAAAAAAAOKLHFCAA 2013-07-20 1362 5925 455 2013 6 7 20 3 2013 455 5925 Saturday N 2013Q3 N Y 2456655 2456475 2456403 2456129 N N N NN
2481780 AAAAAAAAEHONFCAA 2082-10-12 2193 9537 732 2082 1 10 12 4 2082 732 9537 Monday N 2082Q4 N N 2482041 2481769 2481688 2481415 N N N NN
蜂房>描述store_sales;
ss_sold_date_sk int
ss_sold_time_sk int
ss_item_sk int
ss_customer_sk int
ss_cdemo_sk int
ss_hdemo_sk int
ss_addr_sk int
ss_store_sk int
ss_promo_sk int
ss_ticket_number int
ss_quantity int
ss_wholesale_cost decimal(7,2)
ss_list_price decimal(7,2)
ss_sales_price decimal(7,2)
ss_ext_discount_amt decimal(7,2)
ss_ext_sales_price decimal(7,2)
ss_ext_wholesale_cost decimal(7,2)
ss_ext_list_price decimal(7,2)
ss_ext_tax decimal(7,2)
ss_coupon_amt decimal(7,2)
ss_net_paid decimal(7,2)
ss_net_paid_inc_tax decimal(7,2)
ss_net_profit decimal(7,2)
蜂房>描述项目;
i_item_sk int
i_item_id string
i_rec_start_date date
i_rec_end_date date
i_item_desc string
i_current_price decimal(7,2)
i_wholesale_cost decimal(7,2)
i_brand_id int
i_brand string
i_class_id int
i_class string
i_category_id int
i_category string
i_manufact_id int
i_manufact string
i_size string
i_formulation string
i_color string
i_units string
i_container string
i_manager_id int
i_product_name string
蜂房>描述date_dim;
d_date_sk int
d_date_id string
d_date date
d_month_seq int
d_week_seq int
d_quarter_seq int
d_year int
d_dow int
d_moy int
d_dom int
d_qoy int
d_fy_year int
d_fy_quarter_seq int
d_fy_week_seq int
d_day_name string
d_quarter_name string
d_holiday string
d_weekend string
d_following_holiday string
d_first_dom int
d_last_dom int
d_same_day_ly int
d_same_day_lq int
d_current_day string
d_current_week string
d_current_month string
d_current_quarter string
d_current_year string