Question

我正在使用：

Cloudera Manager Free Edition: 4.5.1
Cloudera Hadoop Distro: CDH 4.2.0-1.cdh4.2.0.p0.10 (Parcel)
Hive Metastore with cloudera manager embedded PostgreSQL database.

我的cloudera管理器在一台单独的计算机上运行，它不属于集群。

使用cloudera manager设置集群后，我开始通过hue + beeswax使用配置单元。

一切都运行良好一段时间，然后所有的Suddden，每当我对具有大量分区（大约14000）的特定表运行任何查询时，查询开始超时：

FAILED: SemanticException org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

当我注意到这一点时，我查看了日志，发现与Hive Metastore的连接超时：

WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect. org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out

看过这个，我觉得蜂巢式Metastore存在问题。所以我查看了hive Metastore的日志并发现了java.lang.OutOfMemoryErrors：

/var/log/hive/hadoop-cmf-hive1-HIVEMETASTORE-hci-cdh01.hcinsight.net.log.out:

2013-05-07 14:13:08,744 ERROR org.apache.thrift.ProcessFunction: Internal error processing get_partitions_        with_auth
java.lang.OutOfMemoryError: Java heap space
        at sun.reflectH.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.jav        a:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.datanucleus.util.ClassUtils.newInstance(ClassUtils.java:95)
        at org.datanucleus.store.rdbms.sql.expression.SQLExpressionFactory.newLiteralParameter(SQLExpressi        onFactory.java:248)
        at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.getSQLStatementForIterator(RDBMSMapE        ntrySetStore.java:323)
        at org.datanucleus.store.rdbms.scostore.RDBMSMapEntrySetStore.iterator(RDBMSMapEntrySetStore.java:        221)
        at org.datanucleus.sco.SCOUtils.populateMapDelegateWithStoreData(SCOUtils.java:987)
        at org.datanucleus.sco.backed.Map.loadFromStore(Map.java:258)
        at org.datanucleus.sco.backed.Map.keySet(Map.java:509)
        at org.datanucleus.store.fieldmanager.LoadFieldManager.internalFetchObjectField(LoadFieldManager.j        ava:118)
        at org.datanucleus.store.fieldmanager.AbstractFetchFieldManager.fetchObjectField(AbstractFetchFiel        dManager.java:114)
        at org.datanucleus.state.AbstractStateManager.replacingObjectField(AbstractStateManager.java:1183)
        at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.ja        va)
        at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceFields(MStorageDescriptor.j        ava)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2860)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2879)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsInFetchPlan(JDOStateManagerImpl.java:16        47)
        at org.datanucleus.store.fieldmanager.LoadFieldManager.processPersistable(LoadFieldManager.java:63        )
        at org.datanucleus.store.fieldmanager.LoadFieldManager.internalFetchObjectField(LoadFieldManager.j        ava:84)
        at org.datanucleus.store.fieldmanager.AbstractFetchFieldManager.fetchObjectField(AbstractFetchFiel        dManager.java:104)
        at org.datanucleus.state.AbstractStateManager.replacingObjectField(AbstractStateManager.java:1183)
        at org.apache.hadoop.hive.metastore.model.MPartition.jdoReplaceField(MPartition.java)
        at org.apache.hadoop.hive.metastore.model.MPartition.jdoReplaceFields(MPartition.java)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2860)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2879)
        at org.datanucleus.jdo.state.JDOStateManagerImpl.loadFieldsInFetchPlan(JDOStateManagerImpl.java:16        47)
        at org.datanucleus.ObjectManagerImpl.performDetachAllOnTxnEndPreparation(ObjectManagerImpl.java:35        52)
        at org.datanucleus.ObjectManagerImpl.preCommit(ObjectManagerImpl.java:3291)
        at org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:369)
        at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:256)

此时，hive Metastore会关闭并重新启动：

2013-05-07 14:39:40,576 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Shutting down hive metastore.
2013-05-07 14:41:09,979 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: Starting hive metastore on po        rt 9083

现在，为了解决这个问题，我已经改变了hive Metastore服务器和蜂蜡服务器的最大堆大小：

1. Hive/Hive Metastore Server(Base)/Resource Management/Java Heap Size of Metastore Server : 2 GiB (First thing I did.)
2. Hue/Beeswax Server(Base)/Resource Management/Java Heap Size of Beeswax Server : 2 GiB (After reading some groups posts and stuff online, I tried this as well.)

上面两个步骤中的任何一个似乎都没有帮助，因为我继续在hive Metastore日志中看到OOME。

然后我注意到实际的Metastore'数据库'正在作为我的cloudera管理器的一部分运行，我想知道PostgreSQL进程是否内存不足。我找到了增加该进程的Java堆大小的方法，并且发现了很少的文档。

我想知道你们其中一个人是否可以帮助我解决这个问题。

我应该增加嵌入式数据库的java堆大小吗？如果是这样，我会在哪里这样做？

还有其他我不想要的东西吗？

谢谢！

Answer 1

您是否尝试过以下操作。

 'SET hive.metastore.client.socket.timeout=300;'

这解决了我的问题。让我知道它是怎么回事。

Cloudera Manager嵌入式PostgreSQL Hive Metastore服务器OutOfMemoryError问题

1 个答案: