Question

我有一个使用JobClient提交作业的mapreduce代码。我一直得到这个空指针错误堆栈

12/12/10 12:42:44 INFO mapred.LocalJobRunner: OutputCommitter set in config null    
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.<init>(JobClient.java:226)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:924)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:844)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:844)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:818)
        at FTPIF.run(FTPIF.java:193)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at FTPIF.main(FTPIF.java:273)

让我犯这个错误的代码是

JobClient j = new JobClient();
        j.init(conf);
        RunningJob check = j.submitJob(conf);

有什么想法吗？

Answer 1

我也有这个问题。但使用j.runJob(conf);效果很好。您也可以使用静态函数JobClient.runJob(conf);。我不知道为什么，然后我读了JobClient的源代码并找到了静态函数runJob的实现：

public static RunningJob runJob(JobConf job) throws IOException {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    try {
        if (!jc.monitorAndPrintJob(job, rj)) {
            throw new IOException("Job failed!");
        }
    } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
    }
    return rj;
}

所以我以这种方式使用submitJob函数：

JobClient jcli = new JobClient(jconf);
RunningJob rJob = jcli.submitJob(jconf);
while (true){
    Thread.sleep(5000);
    System.out.println(rJob);
    if (rJob.isComplete())
        break;
}

我工作!!也许这对你也很有用。

Answer 2

我从来没有想过为什么我会点击空指针。设置j.setConf（conf）也不起作用。所以我用runJob来代替正在运行的工作。

RunningJob job= JobClient.runJob(conf);

如何设置OutputCommitter配置？

2 个答案: