在hadoop上运行简单的应用程序

时间:2013-12-14 18:01:11

标签: java hadoop nullpointerexception processbuilder

我正在使用此hadoop mapreduce example application

运行command
hadoop jar e:/Apps/WordCount/hadoop-mapreduce-examples-2.2.0.jar wordcount /users/pavel/wordcount/input /users/pavel/wordcount/output7 

我在日志中看到此错误:

13/12/14 12:54:17 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
13/12/14 12:54:17 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
13/12/14 12:54:20 INFO input.FileInputFormat: Total input paths to process : 2
13/12/14 12:54:20 INFO mapreduce.JobSubmitter: number of splits:2
13/12/14 12:54:20 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/12/14 12:54:20 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
13/12/14 12:54:20 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/12/14 12:54:20 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/12/14 12:54:20 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/12/14 12:54:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1979664780_0001
13/12/14 12:54:21 WARN conf.Configuration: file:/tmp/hadoop-Pavel/mapred/staging/Pavel1979664780/.staging/job_local1979664780_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
13/12/14 12:54:21 WARN conf.Configuration: file:/tmp/hadoop-Pavel/mapred/staging/Pavel1979664780/.staging/job_local1979664780_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
13/12/14 12:54:22 WARN conf.Configuration: file:/tmp/hadoop-Pavel/mapred/local/localRunner/Pavel/job_local1979664780_0001/job_local1979664780_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
13/12/14 12:54:22 WARN conf.Configuration: file:/tmp/hadoop-Pavel/mapred/local/localRunner/Pavel/job_local1979664780_0001/job_local1979664780_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
13/12/14 12:54:22 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
13/12/14 12:54:22 INFO mapreduce.Job: Running job: job_local1979664780_0001
13/12/14 12:54:22 INFO mapred.LocalJobRunner: OutputCommitter set in config null
13/12/14 12:54:22 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
13/12/14 12:54:22 INFO mapred.LocalJobRunner: Waiting for map tasks
13/12/14 12:54:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1979664780_0001_m_000000_0
13/12/14 12:54:22 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
13/12/14 12:54:23 INFO mapreduce.Job: Job job_local1979664780_0001 running in uber mode : false
13/12/14 12:54:23 INFO mapreduce.Job:  map 0% reduce 0%
13/12/14 12:55:13 INFO mapred.LocalJobRunner: Starting task: attempt_local1979664780_0001_m_000001_0
13/12/14 12:55:13 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
13/12/14 12:57:14 INFO mapred.LocalJobRunner: Map task executor complete.
13/12/14 12:57:14 WARN mapred.LocalJobRunner: job_local1979664780_0001
java.lang.Exception: java.lang.NullPointerException
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403)
Caused by: java.lang.NullPointerException
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:441)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:404)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
        at org.apache.hadoop.yarn.util.WindowsBasedProcessTree.isAvailable(WindowsBasedProcessTree.java:57)
        at org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getResourceCalculatorProcessTree(ResourceCalculatorProcessTree.java:153)
        at org.apache.hadoop.mapred.Task.initialize(Task.java:579)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
13/12/14 12:57:15 INFO mapreduce.Job: Job job_local1979664780_0001 failed with state FAILED due to: NA
13/12/14 12:57:15 INFO mapreduce.Job: Counters: 0

我尝试调试此问题并注意到无效的命令列表从ProcessBuilder object传递给Shell objectString[2]: {null, 'help'}null是问题的原因。 但我对如何解决这个问题没有任何想法......你能帮忙吗?

环境: Windows 8,Java 6.应用程序已从cygwin启动。

0 个答案:

没有答案