Question

我在启动NullPointerException工作时得到MapReduce。它被SerializationFactory的{{1}}方法抛出。我使用的是自定义getSerializer()，InputSplit，InputFormat和RecordReader值类。

我知道错误是在我的MapReduce类创建拆分后但在创建InputFormat之前抛出的。据我所知，它是在“清理临时区域”消息后直接发生的。

通过检查堆栈跟踪指示的位置中的Hadoop源，看起来当RecordReader收到空getSerialization()指针时发生错误。 JobClient的Class<T>调用这样的方法：

writeNewSplits()

因此，我假设在我的自定义Serializer<T> serializer = factory.getSerializer((Class<T>) split.getClass());对象上调用getClass()时，它返回一个InputSplit指针，但这只是令人费解。有什么想法吗？

错误的完整堆栈跟踪如下：

12/06/24 14:26:49 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:54310/tmp/hadoop-s3cur3/mapred/staging/s3cur3/.staging/job_201206240915_0035
Exception in thread "main" java.lang.NullPointerException
    at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
    at org.apache.hadoop.mapreduce.split.JobSplitWriter.writeNewSplits(JobSplitWriter.java:123)
    at org.apache.hadoop.mapreduce.split.JobSplitWriter.createSplitFiles(JobSplitWriter.java:74)
    at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:968)
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)
    at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
    at edu.cs.illinois.cogcomp.hadoopinterface.infrastructure.CuratorJob.start(CuratorJob.java:94)
    at edu.cs.illinois.cogcomp.hadoopinterface.HadoopInterface.main(HadoopInterface.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

谢谢！

编辑：我的自定义InputSplit的代码如下：

null

Answer 1

InputSplit不扩展Writable，你需要显式声明你的输入split实现了Writable

调用InputSplit的getClass（）时来自Hadoop的JobSplitWriter / SerializationFactory的NullPointerException

1 个答案: