在两个节点hadoop集群中运行映射减少了作业时出错：获取失败太多，读取任务输出错误http：// localhost：50060 / tasklog

时间：2012-12-20 16:29:03

标签： hadoop mapreduce

我已经设置了一个双节点hadoop集群。我启动了hadoop文件系统并映射了缩减后的守护进程而没有错误，并验证它们是在主服务器和从服务器上运行的。我可以使用命令bin / hadoop dfs -getmerge hdfs：//my.domain.com：54310 / user / wordcount / sunzi.txt / tmp / wordcount从主节点和从节点读取输入文件。当我运行map reduce作业时，我看到输出中的错误。作业最终完成但是reduce部分需要很长时间，并且每次打印错误时它都会继续返回到map任务。我的站点配置文件引用了master的dns名称，所以我不知道为什么这个作业试图从'localhost'读取任务输出

12/12/20 10:47:36 INFO input.FileInputFormat: Total input paths to process : 7
12/12/20 10:47:36 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/12/20 10:47:36 WARN snappy.LoadSnappy: Snappy native library not loaded
12/12/20 10:47:36 INFO mapred.JobClient: Running job: job_201212201046_0001
12/12/20 10:47:37 INFO mapred.JobClient:  map 0% reduce 0%
12/12/20 10:47:44 INFO mapred.JobClient:  map 42% reduce 0%
12/12/20 10:47:45 INFO mapred.JobClient:  map 57% reduce 0%
12/12/20 10:47:49 INFO mapred.JobClient:  map 71% reduce 0%
12/12/20 10:47:50 INFO mapred.JobClient:  map 100% reduce 0%
12/12/20 10:47:54 INFO mapred.JobClient:  map 100% reduce 2%
12/12/20 10:48:08 INFO mapred.JobClient: Task Id : attempt_201212201046_0001_m_000002_0, Status : FAILED
Too many fetch-failures
12/12/20 10:48:08 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201212201046_0001_m_000002_0&filter=stdout
12/12/20 10:48:08 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201212201046_0001_m_000002_0&filter=stderr
12/12/20 10:48:08 INFO mapred.JobClient: Task Id : attempt_201212201046_0001_m_000001_0, Status : FAILED
Too many fetch-failures
12/12/20 10:48:09 INFO mapred.JobClient:  map 85% reduce 2%
12/12/20 10:48:10 INFO mapred.JobClient:  map 71% reduce 2%
12/12/20 10:48:11 INFO mapred.JobClient:  map 85% reduce 2%
12/12/20 10:48:12 INFO mapred.JobClient:  map 100% reduce 2%
12/12/20 10:48:33 INFO mapred.JobClient:  map 100% reduce 3%
12/12/20 10:48:34 INFO mapred.JobClient:  map 100% reduce 4%
12/12/20 10:48:47 INFO mapred.JobClient: Task Id : attempt_201212201046_0001_m_000003_0, Status : FAILED
Too many fetch-failures

我在任务跟踪器日志中看到了这一点：

2012-12-20 10:51:22,255 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201212201046_0001_m_000005_2. Ignored.
2012-12-20 10:51:22,256 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:49774, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201212201046_0001_m_000005_2, duration: 1870835
2012-12-20 10:51:22,257 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/apollo/jobcache/job_201212201046_0001/attempt_201212201046_0001_m_000005_2/output/file.out.index in any of the configured local directories
2012-12-20 10:51:23,225 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000002_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:26,239 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000002_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:26,372 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000000_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:32,255 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000002_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:32,387 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000000_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:35,401 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201212201046_0001_r_000000_0 0.19047621% reduce > copy (4 of 7 at 0.00 MB/s) > 
2012-12-20 10:51:37,116 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201212201046_0001_m_000005_2,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/apollo/jobcache/job_201212201046_0001/attempt_201212201046_0001_m_000005_2/output/file.out.index in any of the configured local directories
        at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
        at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:4009)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:848)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

1 个答案:

答案 0 :(得分：3)

对于未来的googlers，我能够通过从/ etc / hosts文件中删除以下条目来解决此问题

127.0.0.1 localhost

我不明白为什么Hadoop会尝试使用localhost来开始。

相关问题
最新问题