如何更改Hadoop集群中的最大容器功能

时间:2014-09-09 21:42:32

标签: hadoop hortonworks-data-platform rhadoop

我按照以下说明在HORTONWORKS SANDBOX上安装了RHADOOP: http://www.research.janahang.com/install-rhadoop-on-hortonworks-hdp-2-0/

似乎所有东西都安装正确。但是当我在底部运行测试脚本时,我得到一个错误,似乎 - (所需的REDUCE功能超过群集中支持的最大容器功能。杀死作业.reduceResourceReqt:4096 maxContainerCapability:2250)很可能是我的问题。

如何设置maxcontainercapability?或者解决这个问题?任何帮助都会受到欢迎。感谢

错误输出在这里:

Be sure to run hdfs.init()
14/09/09 14:29:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for     your platform... using builtin-java classes where applicable
14/09/09 14:29:27 WARN hdfs.BlockReaderLocal: The short-circuit local reads feature     cannot be used because libhadoop cannot be loaded.
packageJobJar: [] [/usr/lib/hadoop-mapreduce/hadoop-streaming-2.4.0.2.1.1.0-385.jar]     /tmp/streamjob4407691883964292767.jar tmpDir=null
14/09/09 14:29:29 INFO client.RMProxy: Connecting to ResourceManager at     sandbox.hortonworks.com/192.168.32.128:8050
14/09/09 14:29:29 INFO client.RMProxy: Connecting to ResourceManager at     sandbox.hortonworks.com/192.168.32.128:8050
14/09/09 14:29:31 INFO mapred.FileInputFormat: Total input paths to process : 1
14/09/09 14:29:32 INFO mapreduce.JobSubmitter: number of splits:2
14/09/09 14:29:32 INFO mapreduce.JobSubmitter: Submitting tokens for job:     job_1410297633075_0001
14/09/09 14:29:33 INFO impl.YarnClientImpl: Submitted application     application_1410297633075_0001
14/09/09 14:29:33 INFO mapreduce.Job: The url to track the job:     http://sandbox.hortonworks.com:8088/proxy/application_1410297633075_0001/
14/09/09 14:29:33 INFO mapreduce.Job: Running job: job_1410297633075_0001
14/09/09 14:29:42 INFO mapreduce.Job: Job job_1410297633075_0001 running in uber mode :     false
14/09/09 14:29:42 INFO mapreduce.Job:  map 100% reduce 100%
14/09/09 14:29:43 INFO mapreduce.Job: Job job_1410297633075_0001 failed with state     KILLED due to: MAP capability required is more than the supported max container capability     in the cluster. Killing the Job. mapResourceReqt: 4096 maxContainerCapability:2250
Job received Kill while in RUNNING state.
REDUCE capability required is more than the supported max container capability in the     cluster. Killing the Job. reduceResourceReqt: 4096 maxContainerCapability:2250

14/09/09 14:29:43 INFO mapreduce.Job: Counters: 2
    Job Counters
            Total time spent by all maps in occupied slots (ms)=0
            Total time spent by all reduces in occupied slots (ms)=0    
14/09/09 14:29:43 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce,  :
hadoop streaming failed with error code 1
Calls: wordcount -> mapreduce -> mr
Execution halted
14/09/09 14:29:49 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion     interval = 360 minutes, Emptier interval = 0 minutes.
Moved: 'hdfs://sandbox.hortonworks.com:8020/tmp/file1f937beb4f39' to trash at:     hdfs://sandbox.hortonworks.com:8020/user/root/.Trash/Current

2 个答案:

答案 0 :(得分:1)

要在Hortonworks 2.1上执行此操作,我必须

  1. 将VirtualBox内存从4096增加到8192(不知道是否有必要)
  2. http://my.local.host:8000
  3. 启用Ambari
  4. http://my.local.host:8080
  5. 登录Ambari
  6. 将yarn.nodemanager.resource.memory-mb和yarn.scheduler.maximum-allocation-mb的值从默认值更改为4096
  7. 保存并重启所有内容(通过Ambari)
  8. 这让我超越了所需的能力"错误,但实际的wordcount.R似乎并不想完成。然而,像hdfs.ls(" / data")这样的东西确实有效。

答案 1 :(得分:-2)

这个内存问题并不容易解决,但我切换到Cloudera平台,一切都按预期工作。