map减少作业执行在地图任务上停止67%

时间:2017-12-01 14:29:00

标签: hadoop mapreduce hdinsight

我在eclipse上用java创建了map reduce作业。我为wordcount创建了map reduce工作,它从sql server读取数据(大约7453215记录 - 670MB)并将结果存储回sql server。我在azure上创建了HDInsight集群它有2个头节点和3个工作节点。每个节点有4个内核和14GB RAM.Map减少了本地成功运行的作业但是我在地图上将地图减少作业的jar文件提交到天蓝色的HDInsight集群然后它在67上的地图任务上停止了%。

这是日志,

  

17/12/01 13:23:20 INFO client.AHSProxy:连接到应用程序    headnodehost / 10.0.0.20上的历史服务器:10200 17/12/01 13:23:21 INFO    client.RequestHedgingRMFailoverProxyProvider:寻找活跃的   RM [rm1,rm2] ... 17/12/01 13:23:21 INFO   client.RequestHedgingRMFailoverProxyProvider:找到活跃的RM [rm2]    17/12/01 13:23:21 WARN mapreduce.JobResourceUploader:Hadoop    命令行选项解析未执行。实施工具   使用ToolRunner接口并执行您的应用程序以解决此问题。    17/12/01 13:23:36 INFO mapreduce.JobSubmitter:分裂数:2    17/12/01 13:23:37 INFO mapreduce.JobSubmitter:提交令牌    工作:job_1512119994740_0011 17/12/01 13:23:37 INFO    impl.YarnClientImpl:提交的应用程序    application_1512119994740_0011 17/12/01 13:23:37 INFO mapreduce.Job:    跟踪工作的网址:    http://hn1-hdpclu.53o3id15rwte5en44vyo02sv0h.dx.internal.cloudapp.net:8088/proxy/application_1512119994740_0011/    17/12/01 13:23:37 INFO mapreduce.Job:正在运行的工作:    job_1512119994740_0011 17/12/01 13:23:47 INFO mapreduce.Job:Job    job_1512119994740_0011以超级模式运行:false

     

17/12/01 13:23:47 INFO mapreduce.Job:地图0%减少0%

     

17/12/01 13:24:00 INFO mapreduce.Job:map 33%reduce 0%

     

17/12/01 13:24:06 INFO mapreduce.Job:地图67%减少0%

错误:

  

2017-12-02 07:09:17,697 INFO [CommitterEvent Processor#1]   org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler:   处理事件EventType:TASK_ABORT 2017-12-02 07:09:17,697   WARN [CommitterEvent Processor#1]   org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:输出   abortTask()中的路径为空2017-12-02 07:09:17,699 INFO   [AsyncDispatcher事件处理程序]   org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:   attempt_1512191608534_0003_m_000000_0 TaskAttempt已转换自   FAIL_TASK_CLEANUP到失败2017-12-02 07:09:17,708 INFO [Thread-56]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:1次失败   在节点10.0.0.11 2017-12-02 07:09:17,709 INFO [AsyncDispatcher事件   handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:   attempt_1512191608534_0003_m_000000_1 TaskAttempt已转换自   新的UNASSIGNED 2017-12-02 07:09:17,709 INFO [Thread-56]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:已添加   attempt_1512191608534_0003_m_000000_1列出失败的地图   2017-12-02 07:09:17,721 INFO [AsyncDispatcher事件处理程序]   org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:   attempt_1512191608534_0003_m_000001_0 TaskAttempt已转换自   FAIL_CONTAINER_CLEANUP到FAIL_TASK_CLEANUP 2017-12-02 07:09:17,728   INFO [CommitterEvent Processor#2]   org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler:   处理事件EventType:TASK_ABORT 2017-12-02 07:09:17,728   WARN [CommitterEvent Processor#2]   org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:输出   abortTask()中的路径为空2017-12-02 07:09:17,728 INFO   [AsyncDispatcher事件处理程序]   org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:   attempt_1512191608534_0003_m_000001_0 TaskAttempt已转换自   FAIL_TASK_CLEANUP到失败2017-12-02 07:09:17,729 INFO [Thread-56]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:2次失败   在节点10.0.0.11 2017-12-02 07:09:17,729 INFO [AsyncDispatcher事件   handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:   attempt_1512191608534_0003_m_000001_1 TaskAttempt已转换自   新的UNASSIGNED 2017-12-02 07:09:17,729 INFO [Thread-56]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:已添加   attempt_1512191608534_0003_m_000001_1列出失败的地图   2017-12-02 07:09:18,234 INFO [RMCommunicator Allocator]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:之前   调度:PendingReds:1 ScheduledMaps:2 ScheduledReds:0   AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0   ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:0 2017-12-02 07:09:18,240   INFO [RMCommunicator Allocator]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:   application_1512191608534_0003的getResources():ask = 1 release = 0   newContainers = 0 finishedContainers = 1 resourcelimit = knownNMs = 1 2017-12-02 07:09:18,240 INFO [RMCommunicator   Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:   收到已完成的容器container_1512191608534_0003_01_000002   2017-12-02 07:09:18,240 INFO [RMCommunicator Allocator]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:   重新计算时间表,净空= 2017-12-02   07:09:18,240 INFO [RMCommunicator Allocator]   org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:减少   缓慢启动阈值未达到。 completedMapsForReduceSlowstart 1   2017-12-02 07:09:18,240 INFO [AsyncDispatcher事件处理程序]   org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:   来自attempt_1512191608534_0003_m_000000_0的诊断报告:   ApplicationMaster杀死的容器。集装箱遇难   请求。退出代码为143 Container退出,退出代码为非零   143

0 个答案:

没有答案