Hadoop和Hive o

Question

当我尝试在hive上运行查询时，我收到以下错误：

 FATAL [IPC Server handler 14 on 59490] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1457495991188_0166_m_000000_3 - exited : Java heap space

查询是:(变量$ XXX在执行前被替换）

FROM events_tmp 
INSERT INTO TABLE access PARTITION(day='$TODAY') 
SELECT $CFIELDS WHERE f0='access' AND f11 LIKE '$TODAY%' 
INSERT INTO TABLE access PARTITION(day='$YSTD') 
SELECT $CFIELDS WHERE f0='access' AND f11 LIKE '$YSTD%' 
INSERT INTO TABLE access PARTITION(day='$TMR') 
SELECT $CFIELDS WHERE f0='access' AND f11 LIKE '$TMR%' 
INSERT INTO TABLE downloads PARTITION(day='$TODAY') 
SELECT $IFIELDS WHERE f0='download' AND f36 LIKE '%dl%' AND f5 LIKE '$TODAY%'
INSERT INTO TABLE downloads PARTITION(day='$YSTD')
SELECT $IFIELDS WHERE f0='download' AND f36 LIKE '%dl%' AND f5 LIKE '$YSTD%'
INSERT INTO TABLE downloads PARTITION(day='$TMR') 
SELECT $IFIELDS WHERE f0='download' AND f36 LIKE '%dl%' AND f5 LIKE '$TMR%'

我将导出HADOOP_HEAPSIZE = 4096 添加到 /usr/local/hive/conf/hive-env.sh ，但它不会改变任何内容....服务器上还有足够的可用RAM。

Answer 1

要避免此类错误，您需要在mapred-site.xml文件中更新以下与mapreduce堆大小相关的属性。

<property>
            <name>mapreduce.map.memory.mb</name>
             <value>2048</value>
</property>
<property>
            <name>mapreduce.map.java.opts</name>
            <value>-Xmx2048M</value>
</property>
<property>
            <name>mapreduce.reduce.memory.mb</name>
            <value>2048</value>
</property>
<property>
           <name>mapreduce.reduce.java.opts</name>
           <value>-Xmx2048M</value>
</property>

尝试一下，它应该会有所帮助。