Elastic MapReduce上的Mahout：Java堆空间

时间：2012-04-29 21:33:35

标签： hadoop heap mahout emr

我在Amazon Elastic MapReduce集群上从命令行运行Mahout 0.6，尝试使用canopy-cluster~1500短文档，并且这些作业仍然出现“Error：Java heap space”消息。

根据此处和其他地方的先前问题，我已经找到了我能找到的每个记忆旋钮：

  • conf / hadoop-env.sh：在小型实例上将所有堆空间设置为1.5GB，在大型实例上设置为4GB。

  • conf / mapred-site.xml：添加mapred。{map，reduce} .child.java.opts属性，并将其值设置为-Xmx4000m

  • $ MAHOUT_HOME / bin / mahout：增加JAVA_HEAP_MAX并将MAHOUT_HEAPSIZE设置为6GB（在大型实例上）。

问题仍然存在。我一直在反对这个问题太长时间了 - 有没有人有任何建议？

完整的命令和输出看起来像这样（在大型实例的集群上运行，希望它可以缓解这个问题）：

hadoop@ip-10-80-202-112:~$ mahout-distribution-0.6/bin/mahout canopy -i sparse-data/2010/tf-vectors -o canopy-out/2010 -dm org.apache.mahout.common.distance.TanimotoDistanceMeasure -ow -t1 0.5 -t2 0.005 -cl
run with heapsize 6000
-Xmx6000m
MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Running on hadoop, using HADOOP_HOME=/home/hadoop
No HADOOP_CONF_DIR set, using /home/hadoop/conf 
MAHOUT-JOB: /home/hadoop/mahout-distribution-0.6/mahout-examples-0.6-job.jar
12/04/29 19:50:23 INFO common.AbstractJob: Command line arguments: {--clustering=null, --distanceMeasure=org.apache.mahout.common.distance.TanimotoDistanceMeasure, --endPhase=2147483647, --input=sparse-data/2010/tf-vectors, --method=mapreduce, --output=canopy-out/2010, --overwrite=null, --startPhase=0, --t1=0.5, --t2=0.005, --tempDir=temp}
12/04/29 19:50:24 INFO common.HadoopUtil: Deleting canopy-out/2010
12/04/29 19:50:24 INFO canopy.CanopyDriver: Build Clusters Input: sparse-data/2010/tf-vectors Out: canopy-out/2010 Measure: org.apache.mahout.common.distance.TanimotoDistanceMeasure@a383118 t1: 0.5 t2: 0.0050
12/04/29 19:50:24 INFO mapred.JobClient: Default number of map tasks: null
12/04/29 19:50:24 INFO mapred.JobClient: Setting default number of map tasks based on cluster size to : 24
12/04/29 19:50:24 INFO mapred.JobClient: Default number of reduce tasks: 1
12/04/29 19:50:25 INFO mapred.JobClient: Setting group to hadoop
12/04/29 19:50:25 INFO input.FileInputFormat: Total input paths to process : 1
12/04/29 19:50:25 INFO mapred.JobClient: Running job: job_201204291846_0004
12/04/29 19:50:26 INFO mapred.JobClient:  map 0% reduce 0%
12/04/29 19:50:45 INFO mapred.JobClient:  map 27% reduce 0%
[ ... Continues fine until... ]
12/04/29 20:05:54 INFO mapred.JobClient:  map 100% reduce 99%
12/04/29 20:06:12 INFO mapred.JobClient:  map 100% reduce 0%
12/04/29 20:06:20 INFO mapred.JobClient: Task Id : attempt_201204291846_0004_r_000000_0, Status : FAILED
Error: Java heap space
12/04/29 20:06:41 INFO mapred.JobClient:  map 100% reduce 33%
12/04/29 20:06:44 INFO mapred.JobClient:  map 100% reduce 68%
[.. REPEAT SEVERAL ITERATIONS, UNITL...]
12/04/29 20:37:58 INFO mapred.JobClient: map 100% reduce 0%
12/04/29 20:38:09 INFO mapred.JobClient: Job complete: job_201204291846_0004
12/04/29 20:38:09 INFO mapred.JobClient: Counters: 23
12/04/29 20:38:09 INFO mapred.JobClient: Job Counters 
12/04/29 20:38:09 INFO mapred.JobClient: Launched reduce tasks=4
12/04/29 20:38:09 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=94447
12/04/29 20:38:09 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/04/29 20:38:09 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/04/29 20:38:09 INFO mapred.JobClient: Rack-local map tasks=1
12/04/29 20:38:09 INFO mapred.JobClient: Launched map tasks=1
12/04/29 20:38:09 INFO mapred.JobClient: Failed reduce tasks=1
12/04/29 20:38:09 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=23031
12/04/29 20:38:09 INFO mapred.JobClient: FileSystemCounters
12/04/29 20:38:09 INFO mapred.JobClient: HDFS_BYTES_READ=24100612
12/04/29 20:38:09 INFO mapred.JobClient: FILE_BYTES_WRITTEN=49399745
12/04/29 20:38:09 INFO mapred.JobClient: File Input Format Counters 
12/04/29 20:38:09 INFO mapred.JobClient: Bytes Read=24100469
12/04/29 20:38:09 INFO mapred.JobClient: Map-Reduce Framework
12/04/29 20:38:09 INFO mapred.JobClient: Map output materialized bytes=49374728
12/04/29 20:38:09 INFO mapred.JobClient: Combine output records=0
12/04/29 20:38:09 INFO mapred.JobClient: Map input records=409
12/04/29 20:38:09 INFO mapred.JobClient: Physical memory (bytes) snapshot=2785939456
12/04/29 20:38:09 INFO mapred.JobClient: Spilled Records=409
12/04/29 20:38:09 INFO mapred.JobClient: Map output bytes=118596530
12/04/29 20:38:09 INFO mapred.JobClient: CPU time spent (ms)=83190
12/04/29 20:38:09 INFO mapred.JobClient: Total committed heap usage (bytes)=2548629504
12/04/29 20:38:09 INFO mapred.JobClient: Virtual memory (bytes) snapshot=4584386560
12/04/29 20:38:09 INFO mapred.JobClient: Combine input records=0
12/04/29 20:38:09 INFO mapred.JobClient: Map output records=409
12/04/29 20:38:09 INFO mapred.JobClient: SPLIT_RAW_BYTES=143
Exception in thread "main" java.lang.InterruptedException: Canopy Job failed processing sparse-data/2010/tf-vectors
at org.apache.mahout.clustering.canopy.CanopyDriver.buildClustersMR(CanopyDriver.java:349)
at org.apache.mahout.clustering.canopy.CanopyDriver.buildClusters(CanopyDriver.java:236)
at org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:145)
at org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:109)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.mahout.clustering.canopy.CanopyDriver.main(CanopyDriver.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

2 个答案:

答案 0 :(得分：3)

在正常情况下，您可以通过设置“mapred.map.child.java.opts”和/或“mapred.reduce.child.java.opts”来增加map / reduce子任务的内存分配“-Xmx3g”。

但是，当您在AWS上运行时，您对这些设置的直接控制较少。 Amazon提供了一种在启动时配置EMR集群的机制，称为“引导操作”。

对于内存密集型工作流程，即任何Mahout :)，请查看“MemoryIntensive”引导程序。

http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/Bootstrap.html#PredefinedBootstrapActions_MemoryIntensive

答案 1 :(得分：1)

您的本地Hadoop配置与EMR的运行方式无关，也不会与这些环境变量有关。您必须自己配置EMR，并且没有相应的部分。例如，您的工作记忆取决于您要求的实例类型。

错误并不表示与内存有任何关系。由于某种原因，EMR在等待它完成时中断了工作。它失败了吗？

相关问题
最新问题