mahout中的Java堆空间错误

时间:2014-12-07 09:56:10

标签: java hadoop mapreduce mahout

我已经完成了前面提到的问题,并尝试了那里提到的解决方案。它对我来说没有用。运行将文档目录转换为序列文件时,它是相同的Java堆空间错误。 creating vectors from text

我尝试使用export MAHOUT_HEAPSIZE=10000M更改mahout堆大小。这没有用。在某个地方,我看到它应该检查哪个进程正在消耗内存或者堆空间耗尽了什么。为此,我在运行mahout工作之前运行了jps。这并没有给我提供进程ID或提及here的任何内容。也许我做得不对。

MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath.
Running on hadoop, using /opt/hadoop/2.4.0/bin/hadoop and HADOOP_CONF_DIR=/opt/hadoop/2.4.0/etc/hadoop
MAHOUT-JOB: /opt/mahout/1.0-SNAPSHOT/mahout-examples-1.0-SNAPSHOT-job.jar
14/12/07 10:52:47 INFO common.AbstractJob: Command line arguments: {--charset=[UTF-8], --chunkSize=[64], --endPhase=[2147483647], --fileFilterClass=[org.apache.mahout.text.PrefixAdditionFilter], --input=[/cloudc21/output/part-r-00000], --keyPrefix=[], --method=[mapreduce], --output=[/cloudc21/output2/], --startPhase=[0], --tempDir=[temp]}
OpenJDK 64-Bit Server VM warning: You have loaded library /opt/hadoop/2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/12/07 10:52:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/12/07 10:52:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/12/07 10:52:49 INFO Configuration.deprecation: mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress
14/12/07 10:52:49 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/12/07 10:52:51 INFO client.RMProxy: Connecting to ResourceManager at shark/192.168.1.170:10040
14/12/07 10:52:53 INFO input.FileInputFormat: Total input paths to process : 1
14/12/07 10:52:53 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 2, size left: 0
14/12/07 10:52:53 INFO mapreduce.JobSubmitter: number of splits:1
14/12/07 10:52:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1417970396048_0020
14/12/07 10:52:55 INFO impl.YarnClientImpl: Submitted application application_1417970396048_0020
14/12/07 10:52:55 INFO mapreduce.Job: The url to track the job: http://shark:8088/proxy/application_1417970396048_0020/
14/12/07 10:52:55 INFO mapreduce.Job: Running job: job_1417970396048_0020
14/12/07 10:53:08 INFO mapreduce.Job: Job job_1417970396048_0020 running in uber mode : false
14/12/07 10:53:08 INFO mapreduce.Job:  map 0% reduce 0%
14/12/07 10:53:24 INFO mapreduce.Job:  map 50% reduce 0%
14/12/07 10:53:32 INFO mapreduce.Job:  map 100% reduce 0%
14/12/07 10:53:32 INFO mapreduce.Job: Task Id : attempt_1417970396048_0020_m_000000_0, Status : FAILED
Error: Java heap space
14/12/07 10:53:33 INFO mapreduce.Job:  map 0% reduce 0%
14/12/07 10:53:47 INFO mapreduce.Job:  map 50% reduce 0%
14/12/07 10:53:49 INFO mapreduce.Job: Task Id : attempt_1417970396048_0020_m_000000_1, Status : FAILED
Error: Java heap space
14/12/07 10:53:50 INFO mapreduce.Job:  map 0% reduce 0%
14/12/07 10:54:05 INFO mapreduce.Job:  map 50% reduce 0%
14/12/07 10:54:06 INFO mapreduce.Job:  map 100% reduce 0%
14/12/07 10:54:06 INFO mapreduce.Job: Task Id : attempt_1417970396048_0020_m_000000_2, Status : FAILED
Error: Java heap space
14/12/07 10:54:07 INFO mapreduce.Job:  map 0% reduce 0%
14/12/07 10:54:20 INFO mapreduce.Job:  map 50% reduce 0%
14/12/07 10:54:23 INFO mapreduce.Job:  map 100% reduce 0%
14/12/07 10:54:23 INFO mapreduce.Job: Job job_1417970396048_0020 failed with state FAILED due to: Task failed task_1417970396048_0020_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/12/07 10:54:23 INFO mapreduce.Job: Counters: 9
    Job Counters 
        Failed map tasks=4
        Launched map tasks=4
        Other local map tasks=3
        Rack-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=68579
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=68579
        Total vcore-seconds taken by all map tasks=68579
        Total megabyte-seconds taken by all map tasks=70224896
14/12/07 10:54:23 INFO driver.MahoutDriver: Program took 97221 ms (Minutes: 1.62035)

请建议

0 个答案:

没有答案