Hadoop单节点配置:映射减少期间的分配内存错误

时间:2014-06-28 17:22:39

标签: java linux hadoop mapreduce vagrant

我试图在Ubuntu 14.04 LTS(GNU / Linux 3.13.0-29-通用x86_64)上设置hadoop单节点(v2.4.0)(我使用vagrant和puppet进行所有配置) 。 使用start-dfs.sh格式化Namenode并启动hadoop服务。 用于配置hadoop的文件是here,虚拟机保留4Gb的内存。 当我使用命令运行示例map reduce时:

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep indput output 'dfs[a-z]+'

我收到以下错误:

  

OpenJDK 64位服务器VM警告:信息:   os :: commit_memory(0x00000000e0d3e000,104861696,0)失败;   错误='无法分配内存' (错误= 12)

这是地图缩小输出的转储:

vagrant@vagrant:/usr/local/hadoop$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep indput output 'dfs[a-z]+'
14/06/28 17:19:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/28 17:19:11 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/06/28 17:19:11 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/06/28 17:19:11 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/06/28 17:19:11 INFO input.FileInputFormat: Total input paths to process : 25
14/06/28 17:19:12 INFO mapreduce.JobSubmitter: number of splits:25
14/06/28 17:19:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local41566469_0001
14/06/28 17:19:12 WARN conf.Configuration: file:/usr/local/hadoop/tmp/mapred/staging/vagrant41566469/.staging/job_local41566469_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/06/28 17:19:12 WARN conf.Configuration: file:/usr/local/hadoop/tmp/mapred/staging/vagrant41566469/.staging/job_local41566469_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/06/28 17:19:13 WARN conf.Configuration: file:/usr/local/hadoop/tmp/mapred/local/localRunner/vagrant/job_local41566469_0001/job_local41566469_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/06/28 17:19:13 WARN conf.Configuration: file:/usr/local/hadoop/tmp/mapred/local/localRunner/vagrant/job_local41566469_0001/job_local41566469_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/06/28 17:19:13 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/06/28 17:19:13 INFO mapreduce.Job: Running job: job_local41566469_0001
14/06/28 17:19:13 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/06/28 17:19:13 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/06/28 17:19:13 INFO mapred.LocalJobRunner: Waiting for map tasks
14/06/28 17:19:13 INFO mapred.LocalJobRunner: Starting task: attempt_local41566469_0001_m_000000_0
14/06/28 17:19:13 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/06/28 17:19:13 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/vagrant/indput/log4j.properties:0+11169
14/06/28 17:19:13 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000e0dad000, 104861696, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 104861696 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /usr/local/hadoop-2.4.0/hs_err_pid2828.log

我如何解决这个问题?我是否缺少设置一些配置属性?

由于

再见

詹卢卡

0 个答案:

没有答案