Question

我已在多节点群集（1个namenode和4个datanode）上成功运行Hadoop 2.7.1。但是，当我运行MapReduce作业（来自Hadoop网站的WordCount示例）时，它总是停留在这一点上。

[~@~ hadoop-2.7.1]$ bin/hadoop jar WordCount.jar WordCount /user/inputdata/ /user/outputdata
15/09/30 17:54:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/09/30 17:54:57 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/09/30 17:54:58 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/09/30 17:54:59 INFO input.FileInputFormat: Total input paths to process : 1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: number of splits:1
15/09/30 17:55:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1443606819488_0002
15/09/30 17:55:00 INFO impl.YarnClientImpl: Submitted application application_1443606819488_0002
15/09/30 17:55:00 INFO mapreduce.Job: The url to track the job: http://~~~~:8088/proxy/application_1443606819488_0002/
15/09/30 17:55:00 INFO mapreduce.Job: Running job: job_1443606819488_0002

我是否必须为yarn指定内存？

注意：DataNode硬件非常旧（每个都有1GB RAM）。

感谢您的帮助。谢谢。

Answer 1

数据节点内存（1gb）非常缺乏准备至少1个容器来运行mapper / reducer / am。

您可以尝试使用非常低的值降低yarn-site.xml中的以下容器内存分配值，以便在其上创建容器。

yarn.scheduler.minimum-allocation-mb
yarn.scheduler.maximum-allocation-mb

还尝试在作业配置中减少以下属性值

mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
mapreduce.map.java.opts
mapreduce.reduce.java.opts

MapReduce作业停留在多节点Hadoop-2.7.1集群上

1 个答案: