Question

运行Hive查询时，我的Hadoop 2.3.0群集中的数据节点出现内存不足错误。我应该考虑哪些设置来防止nodemanager关闭？

2014-04-29 12:03:33,505 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Failed to launch container.
java.lang.OutOfMemoryError: Java heap space
    at java.lang.ClassLoader.findLoadedClass0(Native Method)
    at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:932)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
    at javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:112)
    at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
    at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:147)
    at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:265)
    at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2195)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2172)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2089)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:838)
    at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:857)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1876)
    at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:149)
    at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:240)
    at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:332)
    at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:329)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:416)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:329)
    at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:443)
    at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:423)
    at org.apache.hadoop.fs.FileContext.getLocalFSFileContext(FileContext.java:409)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:185)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
2014-04-29 12:03:56,003 FATAL org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Socket Reader #1 for port 8040,5,main] threw an Error.  Shutting down now...

Answer 1

您可能需要增加NodeManager的堆内存，在配置目录的配置文件yarn-env.sh中设置以下环境变量，然后重新启动nodemanger。

export YARN_NODEMANAGER_HEAPSIZE=2048

默认值为1000.即1000MB

Hadoop YARN：无法启动容器

1 个答案: