HADOOP HIVE mr.MapredLocalTask​​(MapredLocalTask​​.java:execute(276)) - 执行失败并退出状态:137

时间:2015-10-07 11:46:57

标签: hadoop mapreduce hive emr elastic-map-reduce

我试图在AWS EMR中使用集群(1个主节点,4个核心节点[每个11.25GB])在hive中运行作业,即时加入(MAP加入)两个表,其中一个表有30万个条目(~11mb),另一个表有近700万条目(注意大表应该是最后一个)。因为它是映射连接,所以第一个小表在分布式缓存中进行散列,在散列了一些条目之后我得到了这个错误。

2015-10-07 11:21:35 Processing rows:    10600000    Hashtable size: 10599999    Memory usage:   5465925664  percentage: 0.143
2015-10-07 11:22:09 Processing rows:    10700000    Hashtable size: 10699999    Memory usage:   6078125280  percentage: 0.159
2015-10-07 11:22:42 Processing rows:    10800000    Hashtable size: 10799999    Memory usage:   2698310976  percentage: 0.071
2015-10-07 11:23:16 Processing rows:    10900000    Hashtable size: 10899999    Memory usage:   3272893216  percentage: 0.086
2015-10-07 11:23:49 Processing rows:    11000000    Hashtable size: 10999999    Memory usage:   3821330720  percentage: 0.10
2015-10-07 11:24:23 Processing rows:    11100000    Hashtable size: 11099999    Memory usage:   4398775280  percentage: 0.115
2015-10-07 11:24:57 Processing rows:    11200000    Hashtable size: 11199999    Memory usage:   5020884784  percentage: 0.132
2015-10-07 11:25:30 Processing rows:    11300000    Hashtable size: 11299999    Memory usage:   5608609488  percentage: 0.147
2015-10-07 11:26:04 Processing rows:    11400000    Hashtable size: 11399999    Memory usage:   6146767072  percentage: 0.161
2015-10-07 11:26:38 Processing rows:    11500000    Hashtable size: 11499999    Memory usage:   2622476744  percentage: 0.069
2015-10-07 11:27:11 Processing rows:    11600000    Hashtable size: 11599999    Memory usage:   3224517304  percentage: 0.084
2015-10-07 11:27:44 Processing rows:    11700000    Hashtable size: 11699999    Memory usage:   3793226384  percentage: 0.099
Execution failed with exit status: 137
Obtaining error information

Task failed!
Task ID:
Stage-4

Logs:

/mnt/var/log/apps/hive.log
FAILED: Execution Error, return code 137 from        org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask 

0 个答案:

没有答案