我在eclipse上用java创建了map reduce作业。我为wordcount创建了map reduce工作,它从sql server读取数据(大约7453215记录 - 670MB)并将结果存储回sql server。我在azure上创建了HDInsight集群它有2个头节点和3个工作节点。每个节点有4个内核和14GB RAM.Map减少了本地成功运行的作业但是我在地图上将地图减少作业的jar文件提交到天蓝色的HDInsight集群然后它在67上的地图任务上停止了%。
这是日志,
17/12/01 13:23:20 INFO client.AHSProxy:连接到应用程序 headnodehost / 10.0.0.20上的历史服务器:10200 17/12/01 13:23:21 INFO client.RequestHedgingRMFailoverProxyProvider:寻找活跃的 RM [rm1,rm2] ... 17/12/01 13:23:21 INFO client.RequestHedgingRMFailoverProxyProvider:找到活跃的RM [rm2] 17/12/01 13:23:21 WARN mapreduce.JobResourceUploader:Hadoop 命令行选项解析未执行。实施工具 使用ToolRunner接口并执行您的应用程序以解决此问题。 17/12/01 13:23:36 INFO mapreduce.JobSubmitter:分裂数:2 17/12/01 13:23:37 INFO mapreduce.JobSubmitter:提交令牌 工作:job_1512119994740_0011 17/12/01 13:23:37 INFO impl.YarnClientImpl:提交的应用程序 application_1512119994740_0011 17/12/01 13:23:37 INFO mapreduce.Job: 跟踪工作的网址: http://hn1-hdpclu.53o3id15rwte5en44vyo02sv0h.dx.internal.cloudapp.net:8088/proxy/application_1512119994740_0011/ 17/12/01 13:23:37 INFO mapreduce.Job:正在运行的工作: job_1512119994740_0011 17/12/01 13:23:47 INFO mapreduce.Job:Job job_1512119994740_0011以超级模式运行:false
17/12/01 13:23:47 INFO mapreduce.Job:地图0%减少0%
17/12/01 13:24:00 INFO mapreduce.Job:map 33%reduce 0%
17/12/01 13:24:06 INFO mapreduce.Job:地图67%减少0%
错误:
2017-12-02 07:09:17,697 INFO [CommitterEvent Processor#1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: 处理事件EventType:TASK_ABORT 2017-12-02 07:09:17,697 WARN [CommitterEvent Processor#1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:输出 abortTask()中的路径为空2017-12-02 07:09:17,699 INFO [AsyncDispatcher事件处理程序] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1512191608534_0003_m_000000_0 TaskAttempt已转换自 FAIL_TASK_CLEANUP到失败2017-12-02 07:09:17,708 INFO [Thread-56] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:1次失败 在节点10.0.0.11 2017-12-02 07:09:17,709 INFO [AsyncDispatcher事件 handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1512191608534_0003_m_000000_1 TaskAttempt已转换自 新的UNASSIGNED 2017-12-02 07:09:17,709 INFO [Thread-56] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:已添加 attempt_1512191608534_0003_m_000000_1列出失败的地图 2017-12-02 07:09:17,721 INFO [AsyncDispatcher事件处理程序] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1512191608534_0003_m_000001_0 TaskAttempt已转换自 FAIL_CONTAINER_CLEANUP到FAIL_TASK_CLEANUP 2017-12-02 07:09:17,728 INFO [CommitterEvent Processor#2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: 处理事件EventType:TASK_ABORT 2017-12-02 07:09:17,728 WARN [CommitterEvent Processor#2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:输出 abortTask()中的路径为空2017-12-02 07:09:17,728 INFO [AsyncDispatcher事件处理程序] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1512191608534_0003_m_000001_0 TaskAttempt已转换自 FAIL_TASK_CLEANUP到失败2017-12-02 07:09:17,729 INFO [Thread-56] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:2次失败 在节点10.0.0.11 2017-12-02 07:09:17,729 INFO [AsyncDispatcher事件 handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1512191608534_0003_m_000001_1 TaskAttempt已转换自 新的UNASSIGNED 2017-12-02 07:09:17,729 INFO [Thread-56] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:已添加 attempt_1512191608534_0003_m_000001_1列出失败的地图 2017-12-02 07:09:18,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:之前 调度:PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0 HostLocal:0 RackLocal:0 2017-12-02 07:09:18,240 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: application_1512191608534_0003的getResources():ask = 1 release = 0 newContainers = 0 finishedContainers = 1 resourcelimit = knownNMs = 1 2017-12-02 07:09:18,240 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 收到已完成的容器container_1512191608534_0003_01_000002 2017-12-02 07:09:18,240 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 重新计算时间表,净空= 2017-12-02 07:09:18,240 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:减少 缓慢启动阈值未达到。 completedMapsForReduceSlowstart 1 2017-12-02 07:09:18,240 INFO [AsyncDispatcher事件处理程序] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: 来自attempt_1512191608534_0003_m_000000_0的诊断报告: ApplicationMaster杀死的容器。集装箱遇难 请求。退出代码为143 Container退出,退出代码为非零 143