Question

命令：./ crawl / urls / mydir XXXXX 2

当我在Hadoop-2.5.1和Nutch-2.2.1中运行此命令时，我得到的错误信息如下：

14/10/07 19:58:10 INFO mapreduce.Job：正在运行的工作：job_1411692996443_0016
14/10/07 19:58:17 INFO mapreduce.Job：作业job_1411692996443_0016在超级模式下运行：false 14/10/07 19:58:17 INFO mapreduce.Job：地图0％减少0％
14/10/07 19:58:21 INFO mapreduce.Job：任务ID：attempt_1411692996443_0016_m_000000_0，状态：未通过 错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但是预期了类
14/10/07 19:58:26 INFO mapreduce.Job：任务ID：attempt_1411692996443_0016_m_000000_1，状态：未通过 错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但是预期了类 14/10/07 19:58:31 INFO mapreduce.Job：任务ID：attempt_1411692996443_0016_m_000000_2，状态：未通过 错误：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但是预期了类 14/10/07 19:58:36 INFO mapreduce.Job：地图100％减少0％ 14/10/07 19:58:36 INFO mapreduce.Job：作业job_1411692996443_0016因状态失败而失败，原因是：任务失败task_1411692996443_0016_m_000000
任务失败，作业失败。 failedMaps：1次失败减少：0
14/10/07 19:58:36 INFO mapreduce.Job：专柜：12

Job Counters 
    Failed map tasks=4
    Launched map tasks=4
    Other local map tasks=3
    Data-local map tasks=1
    Total time spent by all maps in occupied slots (ms)=11785
    Total time spent by all reduces in occupied slots (ms)=0
    Total time spent by all map tasks (ms)=11785
    Total vcore-seconds taken by all map tasks=11785
    Total megabyte-seconds taken by all map tasks=12067840
Map-Reduce Framework
    CPU time spent (ms)=0
    Physical memory (bytes) snapshot=0
    Virtual memory (bytes) snapshot=0

14/10/07 19:58:36 ERROR crawl.InjectorJob：InjectorJob：java.lang.RuntimeException：job failed：name = [/ mydir] inject / urls，jobid = job_1411692996443_0016

at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:55)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:233)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:251)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:273)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Answer 1

可能你正在使用用Hadoop 1（来自maven repo？）编译的Gora（或其他）。您可以下载Gora（0.5？）并使用Hadoop 2构建它。

也许这只是一系列问题中的第一个麻烦。请通知我们您未来的步骤。

Answer 2

我在带有hadoop 2.4.0的nutch 2.x上有类似的错误

使用hadoop 2.5.1依赖项（ivy）重新编译nutch并排除所有hadoop 1.x依赖项 - 你可以在lib中找到它们 - 可能是hadoop-core。

Hadoop-2.5.1 + Nutch-2.2.1：找到接口org.apache.hadoop.mapreduce.TaskAttemptContext，但是类是预期的

2 个答案: