一个常见问题,但我仍然无法解决它。
wc mapred作业在地图上停留并减少0%。
以下是节点管理器日志
----------------------------------------------------- Log start -----------------------------------------
015-11-23 10:15:18,789 INFO org.apache.spark.network.yarn.YarnShuffleService: Started YARN shuffle service for Spark on port 7337. Authentication is not enabled. Registered executor file is /yarn/nm/registeredExecutors.ldb
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@6b2d2828
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: false
2015-11-23 10:15:18,815 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=1024 virtual-memory=2151 virtual-cores=2
2015-11-23 10:15:18,816 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService waiting for pending aggregation during exit
2015-11-23 10:15:18,819 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2015-11-23 10:15:18,820 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2015-11-23 10:15:18,821 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2015-11-23 10:15:18,821 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
--------------------------------Log End ----------------------------------------
我不确定以下是否是主要原因:
ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null
我更新了纱线网站和mapred网站xml资源mb详细信息;但得到同样的错误。
任何建议都表示赞赏.b还有谁能让我知道如何进行伪簇刷新?