是什么让hadoop程序运行得非常慢?

时间:2016-07-11 09:09:37

标签: java hadoop

07/15/16更新:它再次阻碍了......可能是因为我不小心删除了一些CleanMyMac,但我不知道我删除了什么。在过去的两天里,情况很好,所以必须是操作系统生成的东西。

07/13/16更新:在离开我的电脑两天后,这个问题已经不知何故消失了,但我仍然不确定为什么会这样。我发现我在tomcat中有一个死会话,不知道这是否相关。

原始问题:我正在做一些apache nutch开发,到上周五结束时,它一切顺利。但突然之间,我的程序今天运行速度非常慢,我在过去两天甚至没有触及我的工作文件,可能出现什么问题?

我尝试运行单节点hadoop示例,WordCount.java,apache hadoop wiki上提供的示例,输入文件只包含两个单词。在完成之前它跑了 2~3分钟

日志文件相当简单,我找不到任何可以使用的东西。

2016-07-11 16:54:37,553 WARN  util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-07-11 16:56:38,174 WARN  mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-07-11 16:56:38,180 WARN  mapreduce.JobSubmitter - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2016-07-11 16:56:38,445 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/staging/Nathaniel_Wu1492571083/.staging/job_local1492571083_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-07-11 16:56:38,449 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/staging/Nathaniel_Wu1492571083/.staging/job_local1492571083_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-07-11 16:56:38,608 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/local/localRunner/Nathaniel_Wu/job_local1492571083_0001/job_local1492571083_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-07-11 16:56:38,611 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/local/localRunner/Nathaniel_Wu/job_local1492571083_0001/job_local1492571083_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

经过一段时间的研究,我发现如果我拔掉网络,它的运行速度可以非常快,总运行时间大约是 2~3秒。但是除了时间戳之外,日志几乎是一样的。

2016-07-11 17:01:30,536 WARN  util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-07-11 17:01:30,928 WARN  mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-07-11 17:01:30,932 WARN  mapreduce.JobSubmitter - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2016-07-11 17:01:31,192 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/staging/Nathaniel_Wu986778478/.staging/job_local986778478_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-07-11 17:01:31,196 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/staging/Nathaniel_Wu986778478/.staging/job_local986778478_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2016-07-11 17:01:31,340 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/local/localRunner/Nathaniel_Wu/job_local986778478_0001/job_local986778478_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2016-07-11 17:01:31,342 WARN  conf.Configuration - file:/tmp/hadoop-Nathaniel_Wu/mapred/local/localRunner/Nathaniel_Wu/job_local986778478_0001/job_local986778478_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.

我想这个问题与集群配置有关,但问题是,我根本没有配置,我的所有程序都应该是单节点的。我在Eclipse中运行我的程序,我只是导入了所需的类,并且根本没有编写任何配置文件。

现在我完全迷失了,我是hadoop的新手,请帮忙,这可能会导致这个问题?

顺便说一下,我在OS X上做了所有这些,这有关系吗?

0 个答案:

没有答案