Question

我们正在使用Hive（0.13.0）测试多节点hadoop集群（2.4.0）。群集工作正常，但是当我们在hive中运行查询时，mapred作业总是在本地执行。例如：

如果没有hive-site.xml（事实上，除了默认值之外没有任何配置文件）我们设置了mapred.job.tracker：

hive> SET mapred.job.tracker=192.168.7.183:8032;

并运行查询：

hive> select count(1) from suricata;

Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
OpenJDK 64-Bit Server VM warning: You have loaded library /hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/04/29 12:48:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
Execution log at: /tmp/hadoopuser/hadoopuser_20140429124747_badfcce6-620e-4718-8c3b-e4ef76bdba7e.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2014-04-29 12:48:05,450 null map = 0%,  reduce = 0%
.......
.......
2014-04-29 12:52:26,982 null map = 100%,  reduce = 100%
Ended Job = job_local1983771849_0001
Execution completed successfully
**MapredLocal task succeeded**
OK
266559841
Time taken: 270.176 seconds, Fetched: 1 row(s)

我们缺少什么？

Answer 1

将 hive.exec.mode.local.auto设置为false ，这将禁用Hive中的本地模式执行

Answer 2

对于每个查询，编译器都会生成map-reduce作业的DAG。如果作业以本地模式运行，请检查以下属性：

mapreduce.framework.name=local;
hive.exec.mode.local.auto=false;

如果启用了自动选项，则hive以本地模式运行作业

Total input size < hive.exec.mode.local.auto.inputbytes.max
Total number of map tasks < hive.exec.mode.local.auto.tasks.max
Total number of reduce tasks =< 1 or 0

这些选项可从0.7

获得

Hive始终以本地模式运行mapred作业

2 个答案: