在hive map side join中,作业停留在Stage-2 map = 82%。我不知道这些是怎么发生的,以及如何检查问题
属性表聚为16个桶,其中有1430467条记录; target_so_many_bucket表聚集成160个桶,并且有
21290383记录在其中。谢谢你的帮助。 HQL是select count(a.uid) from property a join target_so_many_bucket b on a.uid = b.uid ;
日志
Query ID = hive_20170119100909_a373245f-0f08-4260-b61f-5489e7f75810
Total jobs = 1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/tez/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/01/19 10:09:34 WARN conf.HiveConf: HiveConf of name hive.optimize.mapjoin.mapreduce does not exist
17/01/19 10:09:34 WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist
17/01/19 10:09:34 WARN conf.HiveConf: HiveConf of name hive.server2.enable.impersonation does not exist
17/01/19 10:09:34 WARN conf.HiveConf: HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not exist
Execution log at: /tmp/hive/hive_20170119100909_a373245f-0f08-4260-b61f-5489e7f75810.log
2017-01-19 10:09:34 Starting to launch local task to process map join; maximum memory = 1065484288
2017-01-19 10:09:39 Dump the side-table for tag: 0 with group count: 39853 into file: file:/tmp/hive/8875196c-7d09-4574-8972-c5680dea011c/hive_2017-01-19_10-09-25_203_6968180663669785974-1/-local-10004/HashTable-Stage-2/MapJoin-mapfile00--.hashtable
2017-01-19 10:09:39 Uploaded 1 File to: file:/tmp/hive/8875196c-7d09-4574-8972-c5680dea011c/hive_2017-01-19_10-09-25_203_6968180663669785974-1/-local-10004/HashTable-Stage-2/MapJoin-mapfile00--.hashtable (8369143 bytes)
2017-01-19 10:09:39 End of local task; Time Taken: 5.274 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1484721570196_0020, Tracking URL = http://ngfec0299-master-instance-c9aphhre.novalocal:8088/proxy/application_1484721570196_0020/
Kill Command = /usr/hdp/current/hadoop-client/bin/hadoop job -kill job_1484721570196_0020
Hadoop job information for Stage-2: number of mappers: 5; number of reducers: 1
2017-01-19 10:09:50,946 Stage-2 map = 0%, reduce = 0%
2017-01-19 10:10:02,695 Stage-2 map = 2%, reduce = 0%, Cumulative CPU 6.76 sec
2017-01-19 10:10:14,898 Stage-2 map = 4%, reduce = 0%, Cumulative CPU 37.36 sec
2017-01-19 10:10:18,081 Stage-2 map = 9%, reduce = 0%, Cumulative CPU 46.85 sec
2017-01-19 10:10:19,137 Stage-2 map = 10%, reduce = 0%, Cumulative CPU 48.05 sec
2017-01-19 10:10:21,240 Stage-2 map = 15%, reduce = 0%, Cumulative CPU 55.56 sec
2017-01-19 10:10:24,490 Stage-2 map = 20%, reduce = 0%, Cumulative CPU 65.69 sec
2017-01-19 10:10:27,677 Stage-2 map = 23%, reduce = 0%, Cumulative CPU 74.55 sec
2017-01-19 10:10:28,739 Stage-2 map = 26%, reduce = 0%, Cumulative CPU 75.83 sec
2017-01-19 10:10:30,890 Stage-2 map = 35%, reduce = 0%, Cumulative CPU 83.37 sec
2017-01-19 10:10:31,937 Stage-2 map = 36%, reduce = 0%, Cumulative CPU 85.14 sec
2017-01-19 10:10:35,102 Stage-2 map = 38%, reduce = 0%, Cumulative CPU 93.57 sec
2017-01-19 10:10:38,309 Stage-2 map = 43%, reduce = 0%, Cumulative CPU 102.52 sec
2017-01-19 10:10:40,625 Stage-2 map = 51%, reduce = 0%, Cumulative CPU 110.85 sec
2017-01-19 10:10:42,868 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 119.23 sec
2017-01-19 10:11:42,966 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 259.97 sec
2017-01-19 10:12:43,605 Stage-2 map = 52%, reduce = 0%, Cumulative CPU 426.06 sec
2017-01-19 10:13:16,089 Stage-2 map = 54%, reduce = 0%, Cumulative CPU 519.26 sec
2017-01-19 10:13:19,246 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 527.98 sec
2017-01-19 10:14:19,856 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 640.96 sec
2017-01-19 10:15:20,771 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 753.82 sec
2017-01-19 10:16:21,438 Stage-2 map = 63%, reduce = 0%, Cumulative CPU 866.49 sec
2017-01-19 10:16:38,249 Stage-2 map = 67%, reduce = 0%, Cumulative CPU 899.24 sec
2017-01-19 10:16:41,367 Stage-2 map = 70%, reduce = 0%, Cumulative CPU 904.62 sec
2017-01-19 10:16:50,877 Stage-2 map = 74%, reduce = 0%, Cumulative CPU 921.44 sec
2017-01-19 10:17:09,072 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 954.76 sec
2017-01-19 10:18:09,438 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 1011.39 sec
2017-01-19 10:19:09,904 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 1068.72 sec
2017-01-19 10:20:10,593 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 1126.53 sec
2017-01-19 10:21:11,442 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 1184.05 sec
2017-01-19 10:22:11,870 Stage-2 map = 82%, reduce = 0%, Cumulative CPU 1241.26 sec
2017-01-19 10:22:49,914 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1279.22 sec
2017-01-19 10:23:50,564 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1336.78 sec
2017-01-19 10:24:51,008 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1394.38 sec
2017-01-19 10:25:51,323 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1452.14 sec
2017-01-19 10:26:52,203 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1511.05 sec
2017-01-19 10:27:52,692 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1568.34 sec
2017-01-19 10:28:53,263 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1626.35 sec
2017-01-19 10:29:53,471 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1684.42 sec
2017-01-19 10:30:53,918 Stage-2 map = 82%, reduce = 27%, Cumulative CPU 1743.69 sec