Question

我正在尝试在Hadoop（2.6.1）上运行Gridmix

我能够在纱线上运行地图减少作业并运行瘤胃来提取迹线以模拟但是我无法完成最后一步并运行gridmix。

如果我检查日志，我发现了一个错误：

{"org.apache.hadoop.mapreduce.jobhistory.TaskAttemptUnsuccessfulCompletion"

{"taskid":"task_1449829849459_0042_m_000000","taskType":"MAP","attemptId":"attempt_1449829849459_0042_m_000000_0","finishTime":1449841158377,"hostname":"simo","port":56154,"rackname":"/default-rack","status":"FAILED",

"error":"Error: 

java.lang.ClassNotFoundException:
  org.apache.hadoop.tools.rumen.ResourceUsageMetrics
  \n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  \n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  \n\tat java.security.AccessController.doPrivileged(Native Method)
  \n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  \n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:425)\n\tat     sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
  \n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  \n\tat java.lang.Class.getDeclaredConstructors0(Native Method)
  \n\tat java.lang.Class.privateGetDeclaredConstructors(Class.java:2595)
  \n\tat java.lang.Class.getConstructor0(Class.java:2895)
  \n\tat java.lang.Class.getDeclaredConstructor(Class.java:2066)
  \n\tat org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
  \n\tat org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:66)
  \n\tat org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
  \n\tat org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
  \n\tat org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
  \n\tat org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
  \n\tat org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
  \n\tat java.security.AccessController.doPrivileged(Native Method)
  \n\tat javax.security.auth.Subject.doAs(Subject.java:415)
  \n\tat rg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
  \n\tat org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
\n",
"counters":
  {"org.apache.hadoop.mapreduce.jobhistory.JhCounters":
    {"name":"COUNTERS",
     "groups":
        [{"name":"org.apache.hadoop.mapreduce.TaskCounter",
          "displayName":"Map-Reduce Framework",
          "counts":[{"name":"CPU_MILLISECONDS","displayName":"CPU time spent (ms)","value":0},
                    {"name":"PHYSICAL_MEMORY_BYTES","displayName":"Physical memory (bytes) snapshot","value":0},{"name":"VIRTUAL_MEMORY_BYTES","displayName":"Virtual memory (bytes) snapshot","value":0}]}]}},"clockSplits":[286,287,287,287,287,287,286,287,287,287,287,287],"cpuUsages":[0,0,0,0,0,0,0,0,0,0,0,0],"vMemKbytes":[0,0,0,0,0,0,0,0,0,0,0,0],"physMemKbytes":[0,0,0,0,0,0,0,0,0,0,0,0]}}}

这个错误对我来说似乎很奇怪，而且我还没有完全掌握它。

我通过命令行运行gridmix：

bin/hadoop jar share/hadoop/tools/lib/hadoop-gridmix-2.6.1.jar iopath trace.json

和bin/hadoop classpath给我：

$ bin/hadoop classpath
/home/simo/hadoop-2.6.1/conf:
/home/simo/hadoop-2.6.1/share/hadoop/common/lib/*:
/home/simo/hadoop-2.6.1/share/hadoop/common/*:
/home/simo/hadoop-2.6.1/share/hadoop/hdfs:
/home/simo/hadoop-2.6.1/share/hadoop/hdfs/lib/*:
/home/simo/hadoop-2.6.1/share/hadoop/hdfs/*:
/home/simo/hadoop-2.6.1/share/hadoop/yarn/lib/*:
/home/simo/hadoop-2.6.1/share/hadoop/yarn/*:
/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/lib/*:
/home/simo/hadoop-2.6.1/share/hadoop/mapreduce/*:
/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/: <- here is hadoop-rumen-2.6.1.jar
/home/simo/hadoop-2.6.1/share/hadoop/tools/lib/*:
/usr/lib/jvm/java-7-openjdk-amd64/lib/:
/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar:
HADOOP_CLASSPATH

应包括瘤胃文件。

Answer 1

异常来自任务，而不是启动gridmix运行的命令。因此，hadoop类路径并不是这个问题的真正原因。您需要将瘤胃罐放到任务类路径上。

如果您拥有群集的管理权限，则可以将rumen jar（及其依赖项）添加到群集安装中，并确保yarn.application.classpath选择它。

或者，您可以将其作为libjar提供，以便可以将其与作业一起发送到群集。您还需要确保所有必需的依赖项都包含在任务类路径中。

Gridmix无法完成所有工作

1 个答案: