达到100%后地图功能无法完成

时间:2015-07-21 12:56:24

标签: hadoop hbase

我遇到了一个奇怪的问题,当我运行我的地图任务时,它在达到100%后仍会再次运行。

这是我得到的控制台日志:

15/07/22 00:50:12 INFO mapred.JobClient:  map 0% reduce 0%
15/07/22 00:50:17 INFO mapred.LocalJobRunner: 
15/07/22 00:50:18 INFO mapred.JobClient:  map 3% reduce 0%
15/07/22 00:50:20 INFO mapred.LocalJobRunner: 
15/07/22 00:50:21 INFO mapred.JobClient:  map 7% reduce 0%
15/07/22 00:50:23 INFO mapred.LocalJobRunner: 
15/07/22 00:50:26 INFO mapred.LocalJobRunner: 
15/07/22 00:50:27 INFO mapred.JobClient:  map 9% reduce 0%
15/07/22 00:50:29 INFO mapred.LocalJobRunner: 
15/07/22 00:50:30 INFO mapred.JobClient:  map 13% reduce 0%
15/07/22 00:50:32 INFO mapred.LocalJobRunner: 
15/07/22 00:50:33 INFO mapred.JobClient:  map 15% reduce 0%
15/07/22 00:50:35 INFO mapred.LocalJobRunner: 
15/07/22 00:50:38 INFO mapred.LocalJobRunner: 
15/07/22 00:50:39 INFO mapred.JobClient:  map 17% reduce 0%
15/07/22 00:50:41 INFO mapred.LocalJobRunner: 
15/07/22 00:50:42 INFO mapred.JobClient:  map 18% reduce 0%
15/07/22 00:50:44 INFO mapred.LocalJobRunner: 
15/07/22 00:50:45 INFO mapred.JobClient:  map 20% reduce 0%
15/07/22 00:50:47 INFO mapred.LocalJobRunner: 
15/07/22 00:50:48 INFO mapred.JobClient:  map 22% reduce 0%
15/07/22 00:50:50 INFO mapred.LocalJobRunner: 
15/07/22 00:50:51 INFO mapred.JobClient:  map 24% reduce 0%
15/07/22 00:50:53 INFO mapred.LocalJobRunner: 
15/07/22 00:50:54 INFO mapred.JobClient:  map 26% reduce 0%
15/07/22 00:50:56 INFO mapred.LocalJobRunner: 
15/07/22 00:50:57 INFO mapred.JobClient:  map 27% reduce 0%
15/07/22 00:50:59 INFO mapred.LocalJobRunner: 
15/07/22 00:51:00 INFO mapred.JobClient:  map 30% reduce 0%
15/07/22 00:51:02 INFO mapred.LocalJobRunner: 
15/07/22 00:51:03 INFO mapred.JobClient:  map 32% reduce 0%
15/07/22 00:51:05 INFO mapred.LocalJobRunner: 
15/07/22 00:51:06 INFO mapred.JobClient:  map 34% reduce 0%
15/07/22 00:51:09 INFO mapred.LocalJobRunner: 
15/07/22 00:51:12 INFO mapred.LocalJobRunner: 
15/07/22 00:51:12 INFO mapred.JobClient:  map 36% reduce 0%
15/07/22 00:51:15 INFO mapred.LocalJobRunner: 
15/07/22 00:51:15 INFO mapred.JobClient:  map 38% reduce 0%
15/07/22 00:51:18 INFO mapred.LocalJobRunner: 
15/07/22 00:51:18 INFO mapred.JobClient:  map 39% reduce 0%
15/07/22 00:51:21 INFO mapred.LocalJobRunner: 
15/07/22 00:51:21 INFO mapred.JobClient:  map 43% reduce 0%
15/07/22 00:51:24 INFO mapred.LocalJobRunner: 
15/07/22 00:51:24 INFO mapred.JobClient:  map 45% reduce 0%
15/07/22 00:51:27 INFO mapred.LocalJobRunner: 
15/07/22 00:51:27 INFO mapred.JobClient:  map 46% reduce 0%
15/07/22 00:51:30 INFO mapred.LocalJobRunner: 
15/07/22 00:51:30 INFO mapred.JobClient:  map 48% reduce 0%
15/07/22 00:51:33 INFO mapred.LocalJobRunner: 
15/07/22 00:51:33 INFO mapred.JobClient:  map 51% reduce 0%
15/07/22 00:51:36 INFO mapred.LocalJobRunner: 
15/07/22 00:51:36 INFO mapred.JobClient:  map 53% reduce 0%
15/07/22 00:51:39 INFO mapred.LocalJobRunner: 
15/07/22 00:51:39 INFO mapred.JobClient:  map 55% reduce 0%
15/07/22 00:51:42 INFO mapred.LocalJobRunner: 
15/07/22 00:51:42 INFO mapred.JobClient:  map 57% reduce 0%
15/07/22 00:51:45 INFO mapred.LocalJobRunner: 
15/07/22 00:51:45 INFO mapred.JobClient:  map 59% reduce 0%
15/07/22 00:51:48 INFO mapred.LocalJobRunner: 
15/07/22 00:51:49 INFO mapred.JobClient:  map 60% reduce 0%
15/07/22 00:51:51 INFO mapred.LocalJobRunner: 
15/07/22 00:51:51 INFO mapred.JobClient:  map 62% reduce 0%
15/07/22 00:51:54 INFO mapred.LocalJobRunner: 
15/07/22 00:51:54 INFO mapred.JobClient:  map 64% reduce 0%
15/07/22 00:51:57 INFO mapred.LocalJobRunner: 
15/07/22 00:51:57 INFO mapred.JobClient:  map 66% reduce 0%
15/07/22 00:52:00 INFO mapred.LocalJobRunner: 
15/07/22 00:52:00 INFO mapred.JobClient:  map 68% reduce 0%
15/07/22 00:52:03 INFO mapred.LocalJobRunner: 
15/07/22 00:52:06 INFO mapred.LocalJobRunner: 
15/07/22 00:52:06 INFO mapred.JobClient:  map 69% reduce 0%
15/07/22 00:52:09 INFO mapred.LocalJobRunner: 
15/07/22 00:52:12 INFO mapred.LocalJobRunner: 
15/07/22 00:52:21 INFO mapred.LocalJobRunner: 
15/07/22 00:52:22 INFO mapred.JobClient:  map 71% reduce 0%
15/07/22 00:52:24 INFO mapred.LocalJobRunner: 
15/07/22 00:52:30 INFO mapred.LocalJobRunner: 
15/07/22 00:52:31 INFO mapred.JobClient:  map 73% reduce 0%
15/07/22 00:52:36 INFO mapred.LocalJobRunner: 
15/07/22 00:52:37 INFO mapred.JobClient:  map 79% reduce 0%
15/07/22 00:52:40 INFO mapred.LocalJobRunner: 
15/07/22 00:52:40 INFO mapred.JobClient:  map 81% reduce 0%
15/07/22 00:52:43 INFO mapred.LocalJobRunner: 
15/07/22 00:52:46 INFO mapred.LocalJobRunner: 
15/07/22 00:52:46 INFO mapred.JobClient:  map 82% reduce 0%
15/07/22 00:52:50 INFO mapred.LocalJobRunner: 
15/07/22 00:52:51 INFO mapred.JobClient:  map 84% reduce 0%
15/07/22 00:52:53 INFO mapred.LocalJobRunner: 
15/07/22 00:52:59 INFO mapred.LocalJobRunner: 
15/07/22 00:53:00 INFO mapred.JobClient:  map 87% reduce 0%
15/07/22 00:53:03 INFO mapred.LocalJobRunner: 
15/07/22 00:53:09 INFO mapred.LocalJobRunner: 
15/07/22 00:53:10 INFO mapred.JobClient:  map 88% reduce 0%
15/07/22 00:53:12 INFO mapred.LocalJobRunner: 
15/07/22 00:53:14 INFO mapred.JobClient:  map 90% reduce 0%
15/07/22 00:53:15 INFO mapred.LocalJobRunner: 
15/07/22 00:53:16 INFO mapred.JobClient:  map 92% reduce 0%
15/07/22 00:53:18 INFO mapred.LocalJobRunner: 
15/07/22 00:53:25 INFO mapred.LocalJobRunner: 
15/07/22 00:53:25 INFO mapred.JobClient:  map 94% reduce 0%
15/07/22 00:53:31 INFO mapred.LocalJobRunner: 
15/07/22 00:53:31 INFO mapred.JobClient:  map 100% reduce 0%
15/07/22 00:53:33 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting
15/07/22 00:53:34 INFO mapred.LocalJobRunner: 
15/07/22 00:53:34 INFO mapred.LocalJobRunner: 
15/07/22 00:53:34 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
15/07/22 00:53:35 INFO mapreduce.TableOutputFormat: Created table instance for test1
15/07/22 00:53:35 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4af2be62
15/07/22 00:53:40 INFO mapred.LocalJobRunner: 
15/07/22 00:53:40 INFO mapred.JobClient:  map 51% reduce 0%
15/07/22 00:53:43 INFO mapred.LocalJobRunner: 
15/07/22 00:53:43 INFO mapred.JobClient:  map 52% reduce 0%
15/07/22 00:53:46 INFO mapred.LocalJobRunner: 
15/07/22 00:53:46 INFO mapred.JobClient:  map 53% reduce 0%
15/07/22 00:53:49 INFO mapred.LocalJobRunner: 
15/07/22 00:53:49 INFO mapred.JobClient:  map 55% reduce 0%
15/07/22 00:53:52 INFO mapred.LocalJobRunner: 
15/07/22 00:53:52 INFO mapred.JobClient:  map 56% reduce 0%
15/07/22 00:53:55 INFO mapred.LocalJobRunner: 
15/07/22 00:53:55 INFO mapred.JobClient:  map 58% reduce 0%
15/07/22 00:53:58 INFO mapred.LocalJobRunner: 
15/07/22 00:53:58 INFO mapred.JobClient:  map 59% reduce 0%
15/07/22 00:54:01 INFO mapred.LocalJobRunner: 
15/07/22 00:54:01 INFO mapred.JobClient:  map 60% reduce 0%
15/07/22 00:54:04 INFO mapred.LocalJobRunner: 
15/07/22 00:54:10 INFO mapred.LocalJobRunner: 
15/07/22 00:54:10 INFO mapred.JobClient:  map 62% reduce 0%
15/07/22 00:54:13 INFO mapred.LocalJobRunner: 
15/07/22 00:54:13 INFO mapred.JobClient:  map 63% reduce 0%
15/07/22 00:54:16 INFO mapred.LocalJobRunner: 
15/07/22 00:54:17 INFO mapred.JobClient:  map 64% reduce 0%
15/07/22 00:54:19 INFO mapred.LocalJobRunner: 
15/07/22 00:54:20 INFO mapred.JobClient:  map 66% reduce 0%
15/07/22 00:54:22 INFO mapred.LocalJobRunner: 
15/07/22 00:54:23 INFO mapred.JobClient:  map 67% reduce 0%
15/07/22 00:54:25 INFO mapred.LocalJobRunner: 
15/07/22 00:54:26 INFO mapred.JobClient:  map 68% reduce 0%
15/07/22 00:54:28 INFO mapred.LocalJobRunner: 
15/07/22 00:54:29 INFO mapred.JobClient:  map 69% reduce 0%
15/07/22 00:54:31 INFO mapred.LocalJobRunner: 
15/07/22 00:54:32 INFO mapred.JobClient:  map 71% reduce 0%
15/07/22 00:54:34 INFO mapred.LocalJobRunner: 
15/07/22 00:54:35 INFO mapred.JobClient:  map 72% reduce 0%
15/07/22 00:54:37 INFO mapred.LocalJobRunner: 
15/07/22 00:54:38 INFO mapred.JobClient:  map 73% reduce 0%
15/07/22 00:54:40 INFO mapred.LocalJobRunner: 
15/07/22 00:54:41 INFO mapred.JobClient:  map 75% reduce 0%
15/07/22 00:54:43 INFO mapred.LocalJobRunner: 
15/07/22 00:54:44 INFO mapred.JobClient:  map 76% reduce 0%
15/07/22 00:54:46 INFO mapred.LocalJobRunner: 
15/07/22 00:54:47 INFO mapred.JobClient:  map 77% reduce 0%
15/07/22 00:54:49 INFO mapred.LocalJobRunner: 
15/07/22 00:54:50 INFO mapred.JobClient:  map 79% reduce 0%
15/07/22 00:54:52 INFO mapred.LocalJobRunner: 
15/07/22 00:54:53 INFO mapred.JobClient:  map 80% reduce 0%
15/07/22 00:54:55 INFO mapred.LocalJobRunner: 
15/07/22 00:54:56 INFO mapred.JobClient:  map 82% reduce 0%
15/07/22 00:54:58 INFO mapred.LocalJobRunner: 
15/07/22 00:54:59 INFO mapred.JobClient:  map 84% reduce 0%
15/07/22 00:55:01 INFO mapred.LocalJobRunner: 
15/07/22 00:55:04 INFO mapred.LocalJobRunner: 
15/07/22 00:55:05 INFO mapred.JobClient:  map 85% reduce 0%
15/07/22 00:55:07 INFO mapred.LocalJobRunner: 
15/07/22 00:55:10 INFO mapred.LocalJobRunner: 
15/07/22 00:55:11 INFO mapred.JobClient:  map 87% reduce 0%
15/07/22 00:55:13 INFO mapred.LocalJobRunner: 
15/07/22 00:55:14 INFO mapred.JobClient:  map 89% reduce 0%
15/07/22 00:55:16 INFO mapred.LocalJobRunner: 
15/07/22 00:55:17 INFO mapred.JobClient:  map 90% reduce 0%
15/07/22 00:55:21 INFO mapred.LocalJobRunner: 
15/07/22 00:55:24 INFO mapred.LocalJobRunner: 
15/07/22 00:55:25 INFO mapred.JobClient:  map 91% reduce 0%
15/07/22 00:55:28 INFO mapred.LocalJobRunner: 
15/07/22 00:55:29 INFO mapred.JobClient:  map 92% reduce 0%
15/07/22 00:55:32 INFO mapred.LocalJobRunner: 
15/07/22 00:55:32 INFO mapred.JobClient:  map 93% reduce 0%
15/07/22 00:55:35 INFO mapred.LocalJobRunner: 
15/07/22 00:55:35 INFO mapred.JobClient:  map 94% reduce 0%
15/07/22 00:55:38 INFO mapred.LocalJobRunner: 
15/07/22 00:55:39 INFO mapred.JobClient:  map 95% reduce 0%
15/07/22 00:55:41 INFO mapred.LocalJobRunner: 
15/07/22 00:55:42 INFO mapred.JobClient:  map 97% reduce 0%
15/07/22 00:55:44 INFO mapred.LocalJobRunner: 
15/07/22 00:55:45 INFO mapred.JobClient:  map 98% reduce 0%
15/07/22 00:55:47 INFO mapred.LocalJobRunner: 
15/07/22 00:55:48 INFO mapred.JobClient:  map 99% reduce 0%
15/07/22 00:55:50 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And is in the process of commiting
15/07/22 00:55:50 INFO mapred.LocalJobRunner: 
15/07/22 00:55:50 INFO mapred.LocalJobRunner: 
15/07/22 00:55:50 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
15/07/22 00:55:51 INFO mapreduce.TableOutputFormat: Created table instance for test1
15/07/22 00:55:51 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@6491a41a
15/07/22 00:55:51 INFO mapred.JobClient:  map 100% reduce 0%
15/07/22 00:55:56 INFO mapred.LocalJobRunner: 
15/07/22 00:55:57 INFO mapred.JobClient:  map 68% reduce 0%
15/07/22 00:55:59 INFO mapred.LocalJobRunner: 
15/07/22 00:56:00 INFO mapred.JobClient:  map 69% reduce 0%
15/07/22 00:56:02 INFO mapred.LocalJobRunner: 
15/07/22 00:56:03 INFO mapred.JobClient:  map 70% reduce 0%
15/07/22 00:56:05 INFO mapred.LocalJobRunner: 
15/07/22 00:56:06 INFO mapred.JobClient:  map 71% reduce 0%
15/07/22 00:56:08 INFO mapred.LocalJobRunner: 
15/07/22 00:56:09 INFO mapred.JobClient:  map 72% reduce 0%
15/07/22 00:56:11 INFO mapred.LocalJobRunner: 
15/07/22 00:56:14 INFO mapred.LocalJobRunner: 
15/07/22 00:56:15 INFO mapred.JobClient:  map 73% reduce 0%
15/07/22 00:56:17 INFO mapred.LocalJobRunner: 
15/07/22 00:56:18 INFO mapred.JobClient:  map 74% reduce 0%
15/07/22 00:56:20 INFO mapred.LocalJobRunner: 
15/07/22 00:56:21 INFO mapred.JobClient:  map 75% reduce 0%
15/07/22 00:56:23 INFO mapred.LocalJobRunner: 
15/07/22 00:56:26 INFO mapred.LocalJobRunner: 
15/07/22 00:56:27 INFO mapred.JobClient:  map 77% reduce 0%
15/07/22 00:56:30 INFO mapred.LocalJobRunner: 
15/07/22 00:56:36 INFO mapred.LocalJobRunner: 
15/07/22 00:56:36 WARN mapred.FileOutputCommitter: Output path is null in cleanup
15/07/22 00:56:36 INFO mapred.JobClient: Job complete: job_local_0001
15/07/22 00:56:36 INFO mapred.JobClient: Counters: 12
15/07/22 00:56:36 INFO mapred.JobClient:   Map-Reduce Framework
15/07/22 00:56:36 INFO mapred.JobClient:     Spilled Records=0
15/07/22 00:56:36 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
15/07/22 00:56:36 INFO mapred.JobClient:     Map input records=129
15/07/22 00:56:36 INFO mapred.JobClient:     SPLIT_RAW_BYTES=351
15/07/22 00:56:36 INFO mapred.JobClient:     Map output records=128
15/07/22 00:56:36 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
15/07/22 00:56:36 INFO mapred.JobClient:     CPU time spent (ms)=0
15/07/22 00:56:36 INFO mapred.JobClient:     Total committed heap usage (bytes)=183218176
15/07/22 00:56:36 INFO mapred.JobClient:   File Input Format Counters 
15/07/22 00:56:36 INFO mapred.JobClient:     Bytes Read=79622144
15/07/22 00:56:36 INFO mapred.JobClient:   FileSystemCounters
15/07/22 00:56:36 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=36253707
15/07/22 00:56:36 INFO mapred.JobClient:     FILE_BYTES_READ=220612775
15/07/22 00:56:36 INFO mapred.JobClient:   File Output Format Counters 
15/07/22 00:56:36 INFO mapred.JobClient:     Bytes Written=0

在配置中,我设置的reducer数为0。

有谁知道我的地图任务会发生什么?

我只是更新我的工作配置:

这是我的工作配置:

Path inputPath = new Path(inputPathName);
        Job job = new Job(conf, NAME + "_" + tableName);
        job.setJarByClass(Importer.class);
        FileInputFormat.setInputPaths(job, inputPath);
        job.setInputFormatClass(SequenceFileInputFormat.class);
        // job.setInputFormatClass(TextInputFormat.class);
        job.setMapperClass(Importer.class);
        // No reducers. Just write straight to table. Call initTableReducerJob
        // because it sets up the TableOutputFormat.
        TableMapReduceUtil.initTableReducerJob(tableName, null, job);
        job.setNumReduceTasks(0);

0 个答案:

没有答案