与Hive CLI相比,Oozie的运行时间更长

时间:2019-04-18 16:39:24

标签: hive hiveql oozie

我目前遇到2个问题:-

  • Oozie作业配置单元SQL需要2个小时才能完成,而在Hive / Beeline CLI中运行相同的SQL大约需要6到7分钟。

  • 在Oozie和Hive / Beeline CLI中运行了相同的SQL,但是即使在两次执行中都使用了相同的hive-site.xml,它们似乎都使用了不同的内存分配。

在Oozie中运行时出现错误,但在具有相同SQL的CLI中运行时没有错误。 hybridhashtable容器的内存分配有所不同。

从下面的日志中,在Oozie中运行时,分配的内存为5000000,但是当我在Hive / Beeline CLI中运行时,分配为10000000,如果内存更大,则分区分配成功了

任何帮助将不胜感激。

日志

|tez.HashTableLoader|: Memory manager allocates 5000000 bytes for the loading hashtable.
|persistence.HashMapWrapper|: Key count from statistics is 1; setting map size to 2
|persistence.HybridHashTableContainer|: Available memory is not enough to create a HybridHashTableContainer!
|persistence.HybridHashTableContainer|: Total available memory: 5000000
|persistence.HybridHashTableContainer|: Estimated small table size: 105
|persistence.HybridHashTableContainer|: Number of hash partitions to be created: 16
|persistence.HybridHashTableContainer|: Total available memory is: 5000000
|persistence.HybridHashTableContainer|: Write buffer size: 524288
|persistence.HybridHashTableContainer|: Using a bloom-1 filter 2 keys of size 8 bytes
|persistence.HybridHashTableContainer|: Each new partition will require memory: 65636
|persistence.HybridHashTableContainer|: Hash partition 0 is created in memory. Total memory usage so far: 65644
|persistence.HybridHashTableContainer|: Hash partition 1 is created in memory. Total memory usage so far: 131280
|persistence.HybridHashTableContainer|: Hash partition 2 is created in memory. Total memory usage so far: 196916
|persistence.HybridHashTableContainer|: Hash partition 3 is created in memory. Total memory usage so far: 262552
|persistence.HybridHashTableContainer|: Hash partition 4 is created in memory. Total memory usage so far: 328188
|persistence.HybridHashTableContainer|: Hash partition 5 is created in memory. Total memory usage so far: 393824
|persistence.HybridHashTableContainer|: Hash partition 6 is created in memory. Total memory usage so far: 459460
|persistence.HybridHashTableContainer|: Hash partition 7 is created in memory. Total memory usage so far: 525096
|persistence.HybridHashTableContainer|: Hash partition 8 is created in memory. Total memory usage so far: 590732
|persistence.HybridHashTableContainer|: Hash partition 9 is created in memory. Total memory usage so far: 656368
|persistence.HybridHashTableContainer|: Hash partition 10 is created in memory. Total memory usage so far: 722004
|persistence.HybridHashTableContainer|: Hash partition 11 is created in memory. Total memory usage so far: 787640
|persistence.HybridHashTableContainer|: Hash partition 12 is created in memory. Total memory usage so far: 853276
|persistence.HybridHashTableContainer|: Hash partition 13 is created in memory. Total memory usage so far: 918912
|persistence.HybridHashTableContainer|: Hash partition 14 is created in memory. Total memory usage so far: 984548
|persistence.HybridHashTableContainer|: Hash partition 15 is created in memory. Total memory usage so far: 1050184
 |persistence.HybridHashTableContainer|: There is not enough memory to allocate 16 hash partitions.
|persistence.HybridHashTableContainer|: Number of partitions created: 16
|persistence.HybridHashTableContainer|: Number of partitions spilled directly to disk on creation: 0

hive-site.xml

 ....
 <property>
  <name>hive.execution.engine</name>
  <value>tez</value>
 </property>
 <property>
   <name>tez.lib.uris</name>
   <value>maprfs:///apps/tez/tez-0.8,maprfs:///apps/tez/tez-0.8/lib</value>
 </property>
 <property>
   <name>hive.tez.container.size</name>
   <value>6144</value>
 </property>
 ...

0 个答案:

没有答案