读/写时出错(Hadoop)

时间:2012-02-11 18:34:41

标签: java hadoop mapreduce hdfs

我按照this教程安装了hadoop。好吧,如果我使用\ usr \ local \ hadoop作为hadoop.tmp.dir,一切正常。因为我在这个分区中的空间很小,所以我试着将这个值设置为/ NEW_partition /(ext4)但是我总是得到一些java错误。我想这是因为hadoop无法写入此分区。我怎样才能使它发挥作用?

:::: EDIT :::: 完整的执行结果。

hadoop@FreeLnx:/usr/local/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount /MY_STORAGE/tmp1/gutnb /MY_STORAGE/tmp1/gutnbou


12/02/12 02:56:00 INFO input.FileInputFormat: Total input paths to process : 3

12/02/12 02:56:00 INFO mapred.JobClient: Running job: job_201202120255_0001

12/02/12 02:56:01 INFO mapred.JobClient:  map 0% reduce 0%

12/02/12 02:56:09 INFO mapred.JobClient: Task Id : attempt_201202120255_0001_m_000004_0, Status : FAILED

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


12/02/12 02:56:09 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000004_0&filter=stdout

12/02/12 02:56:09 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000004_0&filter=stderr

12/02/12 02:56:15 INFO mapred.JobClient: Task Id : attempt_201202120255_0001_m_000004_1, Status : FAILED

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


12/02/12 02:56:15 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000004_1&filter=stdout

12/02/12 02:56:15 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000004_1&filter=stderr

12/02/12 02:56:21 INFO mapred.JobClient: Task Id : attempt_201202120255_0001_m_000004_2, Status : FAILED

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


12/02/12 02:56:21 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000004_2&filter=stdout

12/02/12 02:56:21 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000004_2&filter=stderr

12/02/12 02:56:33 INFO mapred.JobClient: Task Id : attempt_201202120255_0001_m_000003_0, Status : FAILED

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


12/02/12 02:56:33 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000003_0&filter=stdout

12/02/12 02:56:33 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000003_0&filter=stderr

12/02/12 02:56:39 INFO mapred.JobClient: Task Id : attempt_201202120255_0001_m_000003_1, Status : FAILED

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


12/02/12 02:56:39 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000003_1&filter=stdout

12/02/12 02:56:39 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000003_1&filter=stderr

12/02/12 02:56:45 INFO mapred.JobClient: Task Id : attempt_201202120255_0001_m_000003_2, Status : FAILED

java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)

Caused by: java.io.IOException: Task process exit with nonzero status of 126.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)


12/02/12 02:56:45 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000003_2&filter=stdout

12/02/12 02:56:45 WARN mapred.JobClient: Error reading task outputhttp://FreeLnx:50060/tasklog?plaintext=true&attemptid=attempt_201202120255_0001_m_000003_2&filter=stderr

12/02/12 02:56:51 INFO mapred.JobClient: Job complete: job_201202120255_0001

12/02/12 02:56:51 INFO mapred.JobClient: Counters: 4

12/02/12 02:56:51 INFO mapred.JobClient:   Job Counters 

12/02/12 02:56:51 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=24115

12/02/12 02:56:51 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0

12/02/12 02:56:51 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0

12/02/12 02:56:51 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0

1 个答案:

答案 0 :(得分:0)

您可能需要检查已配置的mapred.local.dir目录的权限,默认值为$ {hadoop.tmp.dir} / mapred / local。父目录和所有内容必须由运行TaskTracker守护程序的用户拥有,以便任务能够有效地在其中写入临时数据(以及执行其他操作)。