在沙箱Hadoop(Cloudera5.1 / Hortonworks2.1)之上是否有与R(build 1060)类似的问题? 这似乎是新R / Hadoop的问题,因为在CDH5.0上它可以工作。
代码:
Sys.setenv(HADOOP_CMD="/usr/bin/hadoop")
Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming.jar")
Sys.setenv(JAVA_HOME="/usr/java/jdk1.7.0_55-cloudera")
library(rhdfs)
library(rmr2)
hdfs.init()
## space and word delimiter
map <- function(k,lines) {
words.list <- strsplit(lines, '\\s')
words <- unlist(words.list)
return( keyval(words, 1) )
}
reduce <- function(word, counts) {
keyval(word, sum(counts))
}
wordcount <- function (input, output=NULL) {
mapreduce(input=input, output=output, input.format="text", map=map, reduce=reduce)
}
## variables
hdfs.root <- '/user/cloudera'
hdfs.data <- file.path(hdfs.root, 'scenario_1')
hdfs.out <- file.path(hdfs.root, 'out')
## run mapreduce job
##out <- wordcount(hdfs.data, hdfs.out)
system.time(out <- wordcount(hdfs.data, hdfs.out))
错误:
> system.time(out <- wordcount(hdfs.data, hdfs.out))
packageJobJar: [] [/usr/lib/hadoop-mapreduce/hadoop-streaming-2.3.0-cdh5.1.0.jar] /tmp/streamjob8497498354509963133.jar tmpDir=null
14/09/17 01:49:38 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
14/09/17 01:49:38 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
14/09/17 01:49:39 INFO mapred.FileInputFormat: Total input paths to process : 1
14/09/17 01:49:39 INFO mapreduce.JobSubmitter: number of splits:2
14/09/17 01:49:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1410940439997_0001
14/09/17 01:49:40 INFO impl.YarnClientImpl: Submitted application application_1410940439997_0001
14/09/17 01:49:40 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1410940439997_0001/
14/09/17 01:49:40 INFO mapreduce.Job: Running job: job_1410940439997_0001
14/09/17 01:49:54 INFO mapreduce.Job: Job job_1410940439997_0001 running in uber mode : false
14/09/17 01:49:54 INFO mapreduce.Job: map 100% reduce 100%
14/09/17 01:49:55 INFO mapreduce.Job: Job job_1410940439997_0001 failed with state KILLED due to: MAP capability required is more than the supported max container capability in the cluster. Killing the Job. mapResourceReqt: 4096 maxContainerCapability:1024
Job received Kill while in RUNNING state.
REDUCE capability required is more than the supported max container capability in the cluster. Killing the Job. **reduceResourceReqt: 4096 maxContainerCapability:1024**
14/09/17 01:49:55 INFO mapreduce.Job: Counters: 2
Job Counters
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
14/09/17 01:49:55 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, : hadoop streaming failed with error code 1
Timing stopped at: 3.681 0.695 20.43
似乎问题出在reduceResourceReqt:4096 maxContainerCapability:1024。 我试图改变:yarn-site.xml,但它没有帮助。 :(
请帮助......
答案 0 :(得分:5)
我没有使用过RHadoop。但是我的群集上遇到了类似的问题,这个问题似乎只与MapReduce有关。
此日志中的 maxContainerCapability 是指yarn.scheduler.maximum-allocation-mb
配置的yarn-site.xml
属性。它是可以在任何容器中使用的最大内存量。
日志中的 mapResourceReqt 和 reduceResourceReqt 会引用mapreduce.map.memory.mb
配置的mapreduce.reduce.memory.mb
和mapred-site.xml
属性。它是将在mapreduce中为Mapper或Reducer创建的容器的内存大小。
如果您的Reducer容器的大小设置为大于yarn.scheduler.maximum-allocation-mb
,这似乎就是这种情况,您的作业将被终止,因为它不允许为容器分配如此多的内存。 / p>
在http://[your-resource-manager]:8088/conf检查您的配置,通常应该找到这些值并查看是这种情况。
也许你的新环境将这些值设置为4096 Mb(相当大,Hadoop 2.7.1中的默认值为1024)。
您应该将mapreduce.[map|reduce].memory.mb
值降低到1024,或者如果您有大量内存并且需要大容器,请将yarn.scheduler.maximum-allocation-mb
值提高到4096.只有MapReduce才能创建容器。
我希望这会有所帮助。