我一直在寻找解决方案,却一无所获。也许有人在这里遇到了这样的问题而且有一些想法。
基本上,我使用hadoop尝试做一些mapReduce任务。一个主节点和两个从节点。配置似乎没问题,因为我可以从每个节点ssh到每个节点。我设置了/etc/hosts
和hadoop的配置,例如:mapred-site.xml
,hdfs-site.xml
。
执行任务后,一切似乎都没问题。没有错误或警告,但当我grep WARN
日志文件时,有一些条目让我担心。
在tasktracker-master.log
我找到了:
2013-06-03 14:14:47,525 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:15:00,680 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:17:44,352 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:17:59,182 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:23:49,419 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:23:55,156 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:27:40,619 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:27:46,382 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:34:15,498 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:34:22,103 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:36:38,551 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031434_0002 being deleted.
2013-06-03 14:37:17,724 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031434_0003 being deleted.
2013-06-03 14:59:42,395 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:59:48,158 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.`
同样的事情发生在slave1/slave2
:
tasktracker-slave1.log:
2013-06-03 14:59:34,439 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:59:45,954 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 15:01:29,380 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031459_0001 being deleted.
2013-06-03 15:03:14,848 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031459_0003 being deleted.
tasktracker-slave2.log:
2013-06-03 14:59:37,642 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:59:45,805 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 15:02:29,489 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031459_0002 being deleted.
任何人都看过这些警告并且知道它们是什么意思?
还有一件事,我怎样才能知道任务是否实际上在所有假定的节点master, slave1, slave2
上执行?这些警告是否意味着slave1/slave2
无效?
我希望我已经清楚地描述了我的问题。 谢谢你的帮助。