我遇到了一个奇怪的问题,当我运行我的地图任务时,它在达到100%后仍会再次运行。
这是我得到的控制台日志:
15/07/22 00:50:12 INFO mapred.JobClient: map 0% reduce 0%
15/07/22 00:50:17 INFO mapred.LocalJobRunner:
15/07/22 00:50:18 INFO mapred.JobClient: map 3% reduce 0%
15/07/22 00:50:20 INFO mapred.LocalJobRunner:
15/07/22 00:50:21 INFO mapred.JobClient: map 7% reduce 0%
15/07/22 00:50:23 INFO mapred.LocalJobRunner:
15/07/22 00:50:26 INFO mapred.LocalJobRunner:
15/07/22 00:50:27 INFO mapred.JobClient: map 9% reduce 0%
15/07/22 00:50:29 INFO mapred.LocalJobRunner:
15/07/22 00:50:30 INFO mapred.JobClient: map 13% reduce 0%
15/07/22 00:50:32 INFO mapred.LocalJobRunner:
15/07/22 00:50:33 INFO mapred.JobClient: map 15% reduce 0%
15/07/22 00:50:35 INFO mapred.LocalJobRunner:
15/07/22 00:50:38 INFO mapred.LocalJobRunner:
15/07/22 00:50:39 INFO mapred.JobClient: map 17% reduce 0%
15/07/22 00:50:41 INFO mapred.LocalJobRunner:
15/07/22 00:50:42 INFO mapred.JobClient: map 18% reduce 0%
15/07/22 00:50:44 INFO mapred.LocalJobRunner:
15/07/22 00:50:45 INFO mapred.JobClient: map 20% reduce 0%
15/07/22 00:50:47 INFO mapred.LocalJobRunner:
15/07/22 00:50:48 INFO mapred.JobClient: map 22% reduce 0%
15/07/22 00:50:50 INFO mapred.LocalJobRunner:
15/07/22 00:50:51 INFO mapred.JobClient: map 24% reduce 0%
15/07/22 00:50:53 INFO mapred.LocalJobRunner:
15/07/22 00:50:54 INFO mapred.JobClient: map 26% reduce 0%
15/07/22 00:50:56 INFO mapred.LocalJobRunner:
15/07/22 00:50:57 INFO mapred.JobClient: map 27% reduce 0%
15/07/22 00:50:59 INFO mapred.LocalJobRunner:
15/07/22 00:51:00 INFO mapred.JobClient: map 30% reduce 0%
15/07/22 00:51:02 INFO mapred.LocalJobRunner:
15/07/22 00:51:03 INFO mapred.JobClient: map 32% reduce 0%
15/07/22 00:51:05 INFO mapred.LocalJobRunner:
15/07/22 00:51:06 INFO mapred.JobClient: map 34% reduce 0%
15/07/22 00:51:09 INFO mapred.LocalJobRunner:
15/07/22 00:51:12 INFO mapred.LocalJobRunner:
15/07/22 00:51:12 INFO mapred.JobClient: map 36% reduce 0%
15/07/22 00:51:15 INFO mapred.LocalJobRunner:
15/07/22 00:51:15 INFO mapred.JobClient: map 38% reduce 0%
15/07/22 00:51:18 INFO mapred.LocalJobRunner:
15/07/22 00:51:18 INFO mapred.JobClient: map 39% reduce 0%
15/07/22 00:51:21 INFO mapred.LocalJobRunner:
15/07/22 00:51:21 INFO mapred.JobClient: map 43% reduce 0%
15/07/22 00:51:24 INFO mapred.LocalJobRunner:
15/07/22 00:51:24 INFO mapred.JobClient: map 45% reduce 0%
15/07/22 00:51:27 INFO mapred.LocalJobRunner:
15/07/22 00:51:27 INFO mapred.JobClient: map 46% reduce 0%
15/07/22 00:51:30 INFO mapred.LocalJobRunner:
15/07/22 00:51:30 INFO mapred.JobClient: map 48% reduce 0%
15/07/22 00:51:33 INFO mapred.LocalJobRunner:
15/07/22 00:51:33 INFO mapred.JobClient: map 51% reduce 0%
15/07/22 00:51:36 INFO mapred.LocalJobRunner:
15/07/22 00:51:36 INFO mapred.JobClient: map 53% reduce 0%
15/07/22 00:51:39 INFO mapred.LocalJobRunner:
15/07/22 00:51:39 INFO mapred.JobClient: map 55% reduce 0%
15/07/22 00:51:42 INFO mapred.LocalJobRunner:
15/07/22 00:51:42 INFO mapred.JobClient: map 57% reduce 0%
15/07/22 00:51:45 INFO mapred.LocalJobRunner:
15/07/22 00:51:45 INFO mapred.JobClient: map 59% reduce 0%
15/07/22 00:51:48 INFO mapred.LocalJobRunner:
15/07/22 00:51:49 INFO mapred.JobClient: map 60% reduce 0%
15/07/22 00:51:51 INFO mapred.LocalJobRunner:
15/07/22 00:51:51 INFO mapred.JobClient: map 62% reduce 0%
15/07/22 00:51:54 INFO mapred.LocalJobRunner:
15/07/22 00:51:54 INFO mapred.JobClient: map 64% reduce 0%
15/07/22 00:51:57 INFO mapred.LocalJobRunner:
15/07/22 00:51:57 INFO mapred.JobClient: map 66% reduce 0%
15/07/22 00:52:00 INFO mapred.LocalJobRunner:
15/07/22 00:52:00 INFO mapred.JobClient: map 68% reduce 0%
15/07/22 00:52:03 INFO mapred.LocalJobRunner:
15/07/22 00:52:06 INFO mapred.LocalJobRunner:
15/07/22 00:52:06 INFO mapred.JobClient: map 69% reduce 0%
15/07/22 00:52:09 INFO mapred.LocalJobRunner:
15/07/22 00:52:12 INFO mapred.LocalJobRunner:
15/07/22 00:52:21 INFO mapred.LocalJobRunner:
15/07/22 00:52:22 INFO mapred.JobClient: map 71% reduce 0%
15/07/22 00:52:24 INFO mapred.LocalJobRunner:
15/07/22 00:52:30 INFO mapred.LocalJobRunner:
15/07/22 00:52:31 INFO mapred.JobClient: map 73% reduce 0%
15/07/22 00:52:36 INFO mapred.LocalJobRunner:
15/07/22 00:52:37 INFO mapred.JobClient: map 79% reduce 0%
15/07/22 00:52:40 INFO mapred.LocalJobRunner:
15/07/22 00:52:40 INFO mapred.JobClient: map 81% reduce 0%
15/07/22 00:52:43 INFO mapred.LocalJobRunner:
15/07/22 00:52:46 INFO mapred.LocalJobRunner:
15/07/22 00:52:46 INFO mapred.JobClient: map 82% reduce 0%
15/07/22 00:52:50 INFO mapred.LocalJobRunner:
15/07/22 00:52:51 INFO mapred.JobClient: map 84% reduce 0%
15/07/22 00:52:53 INFO mapred.LocalJobRunner:
15/07/22 00:52:59 INFO mapred.LocalJobRunner:
15/07/22 00:53:00 INFO mapred.JobClient: map 87% reduce 0%
15/07/22 00:53:03 INFO mapred.LocalJobRunner:
15/07/22 00:53:09 INFO mapred.LocalJobRunner:
15/07/22 00:53:10 INFO mapred.JobClient: map 88% reduce 0%
15/07/22 00:53:12 INFO mapred.LocalJobRunner:
15/07/22 00:53:14 INFO mapred.JobClient: map 90% reduce 0%
15/07/22 00:53:15 INFO mapred.LocalJobRunner:
15/07/22 00:53:16 INFO mapred.JobClient: map 92% reduce 0%
15/07/22 00:53:18 INFO mapred.LocalJobRunner:
15/07/22 00:53:25 INFO mapred.LocalJobRunner:
15/07/22 00:53:25 INFO mapred.JobClient: map 94% reduce 0%
15/07/22 00:53:31 INFO mapred.LocalJobRunner:
15/07/22 00:53:31 INFO mapred.JobClient: map 100% reduce 0%
15/07/22 00:53:33 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
15/07/22 00:53:34 INFO mapred.LocalJobRunner:
15/07/22 00:53:34 INFO mapred.LocalJobRunner:
15/07/22 00:53:34 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
15/07/22 00:53:35 INFO mapreduce.TableOutputFormat: Created table instance for test1
15/07/22 00:53:35 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4af2be62
15/07/22 00:53:40 INFO mapred.LocalJobRunner:
15/07/22 00:53:40 INFO mapred.JobClient: map 51% reduce 0%
15/07/22 00:53:43 INFO mapred.LocalJobRunner:
15/07/22 00:53:43 INFO mapred.JobClient: map 52% reduce 0%
15/07/22 00:53:46 INFO mapred.LocalJobRunner:
15/07/22 00:53:46 INFO mapred.JobClient: map 53% reduce 0%
15/07/22 00:53:49 INFO mapred.LocalJobRunner:
15/07/22 00:53:49 INFO mapred.JobClient: map 55% reduce 0%
15/07/22 00:53:52 INFO mapred.LocalJobRunner:
15/07/22 00:53:52 INFO mapred.JobClient: map 56% reduce 0%
15/07/22 00:53:55 INFO mapred.LocalJobRunner:
15/07/22 00:53:55 INFO mapred.JobClient: map 58% reduce 0%
15/07/22 00:53:58 INFO mapred.LocalJobRunner:
15/07/22 00:53:58 INFO mapred.JobClient: map 59% reduce 0%
15/07/22 00:54:01 INFO mapred.LocalJobRunner:
15/07/22 00:54:01 INFO mapred.JobClient: map 60% reduce 0%
15/07/22 00:54:04 INFO mapred.LocalJobRunner:
15/07/22 00:54:10 INFO mapred.LocalJobRunner:
15/07/22 00:54:10 INFO mapred.JobClient: map 62% reduce 0%
15/07/22 00:54:13 INFO mapred.LocalJobRunner:
15/07/22 00:54:13 INFO mapred.JobClient: map 63% reduce 0%
15/07/22 00:54:16 INFO mapred.LocalJobRunner:
15/07/22 00:54:17 INFO mapred.JobClient: map 64% reduce 0%
15/07/22 00:54:19 INFO mapred.LocalJobRunner:
15/07/22 00:54:20 INFO mapred.JobClient: map 66% reduce 0%
15/07/22 00:54:22 INFO mapred.LocalJobRunner:
15/07/22 00:54:23 INFO mapred.JobClient: map 67% reduce 0%
15/07/22 00:54:25 INFO mapred.LocalJobRunner:
15/07/22 00:54:26 INFO mapred.JobClient: map 68% reduce 0%
15/07/22 00:54:28 INFO mapred.LocalJobRunner:
15/07/22 00:54:29 INFO mapred.JobClient: map 69% reduce 0%
15/07/22 00:54:31 INFO mapred.LocalJobRunner:
15/07/22 00:54:32 INFO mapred.JobClient: map 71% reduce 0%
15/07/22 00:54:34 INFO mapred.LocalJobRunner:
15/07/22 00:54:35 INFO mapred.JobClient: map 72% reduce 0%
15/07/22 00:54:37 INFO mapred.LocalJobRunner:
15/07/22 00:54:38 INFO mapred.JobClient: map 73% reduce 0%
15/07/22 00:54:40 INFO mapred.LocalJobRunner:
15/07/22 00:54:41 INFO mapred.JobClient: map 75% reduce 0%
15/07/22 00:54:43 INFO mapred.LocalJobRunner:
15/07/22 00:54:44 INFO mapred.JobClient: map 76% reduce 0%
15/07/22 00:54:46 INFO mapred.LocalJobRunner:
15/07/22 00:54:47 INFO mapred.JobClient: map 77% reduce 0%
15/07/22 00:54:49 INFO mapred.LocalJobRunner:
15/07/22 00:54:50 INFO mapred.JobClient: map 79% reduce 0%
15/07/22 00:54:52 INFO mapred.LocalJobRunner:
15/07/22 00:54:53 INFO mapred.JobClient: map 80% reduce 0%
15/07/22 00:54:55 INFO mapred.LocalJobRunner:
15/07/22 00:54:56 INFO mapred.JobClient: map 82% reduce 0%
15/07/22 00:54:58 INFO mapred.LocalJobRunner:
15/07/22 00:54:59 INFO mapred.JobClient: map 84% reduce 0%
15/07/22 00:55:01 INFO mapred.LocalJobRunner:
15/07/22 00:55:04 INFO mapred.LocalJobRunner:
15/07/22 00:55:05 INFO mapred.JobClient: map 85% reduce 0%
15/07/22 00:55:07 INFO mapred.LocalJobRunner:
15/07/22 00:55:10 INFO mapred.LocalJobRunner:
15/07/22 00:55:11 INFO mapred.JobClient: map 87% reduce 0%
15/07/22 00:55:13 INFO mapred.LocalJobRunner:
15/07/22 00:55:14 INFO mapred.JobClient: map 89% reduce 0%
15/07/22 00:55:16 INFO mapred.LocalJobRunner:
15/07/22 00:55:17 INFO mapred.JobClient: map 90% reduce 0%
15/07/22 00:55:21 INFO mapred.LocalJobRunner:
15/07/22 00:55:24 INFO mapred.LocalJobRunner:
15/07/22 00:55:25 INFO mapred.JobClient: map 91% reduce 0%
15/07/22 00:55:28 INFO mapred.LocalJobRunner:
15/07/22 00:55:29 INFO mapred.JobClient: map 92% reduce 0%
15/07/22 00:55:32 INFO mapred.LocalJobRunner:
15/07/22 00:55:32 INFO mapred.JobClient: map 93% reduce 0%
15/07/22 00:55:35 INFO mapred.LocalJobRunner:
15/07/22 00:55:35 INFO mapred.JobClient: map 94% reduce 0%
15/07/22 00:55:38 INFO mapred.LocalJobRunner:
15/07/22 00:55:39 INFO mapred.JobClient: map 95% reduce 0%
15/07/22 00:55:41 INFO mapred.LocalJobRunner:
15/07/22 00:55:42 INFO mapred.JobClient: map 97% reduce 0%
15/07/22 00:55:44 INFO mapred.LocalJobRunner:
15/07/22 00:55:45 INFO mapred.JobClient: map 98% reduce 0%
15/07/22 00:55:47 INFO mapred.LocalJobRunner:
15/07/22 00:55:48 INFO mapred.JobClient: map 99% reduce 0%
15/07/22 00:55:50 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
15/07/22 00:55:50 INFO mapred.LocalJobRunner:
15/07/22 00:55:50 INFO mapred.LocalJobRunner:
15/07/22 00:55:50 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
15/07/22 00:55:51 INFO mapreduce.TableOutputFormat: Created table instance for test1
15/07/22 00:55:51 INFO mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@6491a41a
15/07/22 00:55:51 INFO mapred.JobClient: map 100% reduce 0%
15/07/22 00:55:56 INFO mapred.LocalJobRunner:
15/07/22 00:55:57 INFO mapred.JobClient: map 68% reduce 0%
15/07/22 00:55:59 INFO mapred.LocalJobRunner:
15/07/22 00:56:00 INFO mapred.JobClient: map 69% reduce 0%
15/07/22 00:56:02 INFO mapred.LocalJobRunner:
15/07/22 00:56:03 INFO mapred.JobClient: map 70% reduce 0%
15/07/22 00:56:05 INFO mapred.LocalJobRunner:
15/07/22 00:56:06 INFO mapred.JobClient: map 71% reduce 0%
15/07/22 00:56:08 INFO mapred.LocalJobRunner:
15/07/22 00:56:09 INFO mapred.JobClient: map 72% reduce 0%
15/07/22 00:56:11 INFO mapred.LocalJobRunner:
15/07/22 00:56:14 INFO mapred.LocalJobRunner:
15/07/22 00:56:15 INFO mapred.JobClient: map 73% reduce 0%
15/07/22 00:56:17 INFO mapred.LocalJobRunner:
15/07/22 00:56:18 INFO mapred.JobClient: map 74% reduce 0%
15/07/22 00:56:20 INFO mapred.LocalJobRunner:
15/07/22 00:56:21 INFO mapred.JobClient: map 75% reduce 0%
15/07/22 00:56:23 INFO mapred.LocalJobRunner:
15/07/22 00:56:26 INFO mapred.LocalJobRunner:
15/07/22 00:56:27 INFO mapred.JobClient: map 77% reduce 0%
15/07/22 00:56:30 INFO mapred.LocalJobRunner:
15/07/22 00:56:36 INFO mapred.LocalJobRunner:
15/07/22 00:56:36 WARN mapred.FileOutputCommitter: Output path is null in cleanup
15/07/22 00:56:36 INFO mapred.JobClient: Job complete: job_local_0001
15/07/22 00:56:36 INFO mapred.JobClient: Counters: 12
15/07/22 00:56:36 INFO mapred.JobClient: Map-Reduce Framework
15/07/22 00:56:36 INFO mapred.JobClient: Spilled Records=0
15/07/22 00:56:36 INFO mapred.JobClient: Virtual memory (bytes) snapshot=0
15/07/22 00:56:36 INFO mapred.JobClient: Map input records=129
15/07/22 00:56:36 INFO mapred.JobClient: SPLIT_RAW_BYTES=351
15/07/22 00:56:36 INFO mapred.JobClient: Map output records=128
15/07/22 00:56:36 INFO mapred.JobClient: Physical memory (bytes) snapshot=0
15/07/22 00:56:36 INFO mapred.JobClient: CPU time spent (ms)=0
15/07/22 00:56:36 INFO mapred.JobClient: Total committed heap usage (bytes)=183218176
15/07/22 00:56:36 INFO mapred.JobClient: File Input Format Counters
15/07/22 00:56:36 INFO mapred.JobClient: Bytes Read=79622144
15/07/22 00:56:36 INFO mapred.JobClient: FileSystemCounters
15/07/22 00:56:36 INFO mapred.JobClient: FILE_BYTES_WRITTEN=36253707
15/07/22 00:56:36 INFO mapred.JobClient: FILE_BYTES_READ=220612775
15/07/22 00:56:36 INFO mapred.JobClient: File Output Format Counters
15/07/22 00:56:36 INFO mapred.JobClient: Bytes Written=0
在配置中,我设置的reducer数为0。
有谁知道我的地图任务会发生什么?
我只是更新我的工作配置:
这是我的工作配置:
Path inputPath = new Path(inputPathName);
Job job = new Job(conf, NAME + "_" + tableName);
job.setJarByClass(Importer.class);
FileInputFormat.setInputPaths(job, inputPath);
job.setInputFormatClass(SequenceFileInputFormat.class);
// job.setInputFormatClass(TextInputFormat.class);
job.setMapperClass(Importer.class);
// No reducers. Just write straight to table. Call initTableReducerJob
// because it sets up the TableOutputFormat.
TableMapReduceUtil.initTableReducerJob(tableName, null, job);
job.setNumReduceTasks(0);