hadoop集群：hadoop流映射任务仅在一台主机上运行而不在从机中运行

时间：2016-12-19 21:28:24

标签： hadoop mapreduce hdfs hadoop-streaming

我有三台机器的hadoop集群：

  1. 一个主服务器（ResourceManager，NameNode，SecondaryNameNode）
  2. 两个奴隶（DataNode，NodeManager）

    3. 我运行带有hadoop流的c ++程序：

    • 接受输入文本文件，其中包含存储在HDFS下的视频名称

      input.txt中：

      video0001.avi

Video0002.avi

      ...

    • 映射器读取每一行（作为键）后，必须复制名称从hdfs输入的视频并将其存储在从机上，然后程序在视频上运行opencv和ffmpeg，然后切换到视频2做同样的事情

    • mapper将视频名称作为键返回，将视频的某些参数作为值
    • 我在所有群集计算机上都有程序
    • 群集设置良好，我可以将文件复制给您
    • 当我在单个节点上运行程序时，它运行良好，但是当我在单个节点上运行程序时，它工作正常，但是当我在三台机器的集群上运行它时，它只能在不使用从机的主机上运行
    • 我在主机上运行此命令：

    hadoop jar /usr/local/lib/hadoop-2.7.3/share/hadoop/tools/lib/hadoop-streaming-2.7.3.jar -input / user / root / input -output / user / root /输出-mapper signature -file signature 

    •   12/20 02:43:51 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
•   16/12/20 02:43:51 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
•   16/12/20 02:43:51 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
•   16/12/20 02:43:52 INFO mapred.FileInputFormat: Total input paths to process : 1
•   16/12/20 02:43:52 INFO mapreduce.JobSubmitter: number of splits:1
•   16/12/20 02:43:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local815523916_0001
•   16/12/20 02:43:54 INFO mapred.LocalDistributedCacheManager: Localized file:/home/master/Desktop/Extract_signature/Prog/signature as file:/app/hadoop/tmp/mapred/local/1482230633565/signature
•   16/12/20 02:43:54 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
•   16/12/20 02:43:54 INFO mapreduce.Job: Running job: job_local815523916_0001
•   16/12/20 02:43:54 INFO mapred.LocalJobRunner: OutputCommitter set in config null
•   16/12/20 02:43:54 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
•   16/12/20 02:43:54 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
•   16/12/20 02:43:55 INFO mapred.LocalJobRunner: Waiting for map tasks
•   16/12/20 02:43:55 INFO mapred.LocalJobRunner: Starting task: attempt_local815523916_0001_m_000000_0
•   16/12/20 02:43:55 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
•   16/12/20 02:43:55 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
•   16/12/20 02:43:55 INFO mapred.MapTask: Processing split: hdfs://Hadoop:54310/user/root/input/input.txt:0+33
•   16/12/20 02:43:55 INFO mapred.MapTask: numReduceTasks: 1
•   16/12/20 02:43:55 INFO mapreduce.Job: Job job_local815523916_0001 running in uber mode : false
•   16/12/20 02:43:55 INFO mapreduce.Job:  map 0% reduce 0%
•   16/12/20 02:44:48 INFO mapred.LocalJobRunner: hdfs://Hadoop:54310/user/root/input/input.txt:0+33 > map
•   16/12/20 02:44:48 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
•   16/12/20 02:44:48 INFO streaming.PipeMapRed: PipeMapRed exec [/home/master/Desktop/Extract_signature/Prog/./signature]
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir
•   16/12/20 02:44:48 INFO Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.local.dir is deprecated. Instead, use mapreduce.cluster.local.dir
•   16/12/20 02:44:48 INFO Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
•   16/12/20 02:44:48 INFO Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
•   16/12/20 02:44:48 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
•   16/12/20 02:44:48 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
•   16/12/20 02:44:49 INFO streaming.PipeMapRed: R/W/S=1/0/0 in:1=1/1 [rec/s] out:0=0/1 [rec/s]

•   16/12/20 02:44:54 INFO mapred.LocalJobRunner: hdfs://Hadoop:54310/user/root/input/input.txt:0+33 > map
•   16/12/20 02:44:54 INFO mapreduce.Job:  map 67% reduce 0%

•   There were 11 warnings (use warnings() to see them)
•   16/12/20 02:47:48 INFO streaming.PipeMapRed: Records R/W=2/2
•   16/12/20 02:47:48 INFO streaming.PipeMapRed: MRErrorThread done
•   16/12/20 02:47:48 INFO streaming.PipeMapRed: mapRedFinished
•   16/12/20 02:47:48 INFO mapred.LocalJobRunner: Records R/W=2/1 > map
•   16/12/20 02:47:48 INFO mapred.MapTask: Starting flush of map output
•   16/12/20 02:47:48 INFO mapred.MapTask: Spilling map output
•   16/12/20 02:47:48 INFO mapred.MapTask: bufstart = 0; bufend = 40; bufvoid = 104857600
•   16/12/20 02:47:48 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
•   16/12/20 02:47:48 INFO mapred.MapTask: Finished spill 0
•   16/12/20 02:47:48 INFO mapred.Task: Task:attempt_local1256877917_0001_m_000000_0 is done. And is in the process of committing
•   16/12/20 02:47:48 INFO mapred.LocalJobRunner: Records R/W=2/2
•   16/12/20 02:47:48 INFO mapred.Task: Task 'attempt_local1256877917_0001_m_000000_0' done.
•   16/12/20 02:47:48 INFO mapred.LocalJobRunner: Finishing task: attempt_local1256877917_0001_m_000000_0
•   16/12/20 02:47:48 INFO mapred.LocalJobRunner: map task executor complete.
•   16/12/20 02:47:48 INFO mapred.LocalJobRunner: Waiting for reduce tasks
•   16/12/20 02:47:48 INFO mapred.LocalJobRunner: Starting task: attempt_local1256877917_0001_r_000000_0
•   16/12/20 02:47:48 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
•   16/12/20 02:47:48 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
•   16/12/20 02:47:49 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@71589312
•   16/12/20 02:47:49 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
•   16/12/20 02:47:49 INFO reduce.EventFetcher: attempt_local1256877917_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
•   16/12/20 02:47:49 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1256877917_0001_m_000000_0 decomp: 46 len: 50 to MEMORY
•   16/12/20 02:47:49 INFO reduce.InMemoryMapOutput: Read 46 bytes from map-output for attempt_local1256877917_0001_m_000000_0
•   16/12/20 02:47:49 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 46, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->46
•   16/12/20 02:47:49 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
•   16/12/20 02:47:49 INFO mapred.LocalJobRunner: 1 / 1 copied.
•   16/12/20 02:47:49 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
•   16/12/20 02:47:49 INFO mapred.Merger: Merging 1 sorted segments
•   16/12/20 02:47:49 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 25 bytes
•   16/12/20 02:47:49 INFO reduce.MergeManagerImpl: Merged 1 segments, 46 bytes to disk to satisfy reduce memory limit
•   16/12/20 02:47:49 INFO reduce.MergeManagerImpl: Merging 1 files, 50 bytes from disk
•   16/12/20 02:47:49 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
•   16/12/20 02:47:49 INFO mapred.Merger: Merging 1 sorted segments
•   16/12/20 02:47:49 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 25 bytes
•   16/12/20 02:47:49 INFO mapred.LocalJobRunner: 1 / 1 copied.
•   16/12/20 02:47:49 INFO mapred.Task: Task:attempt_local1256877917_0001_r_000000_0 is done. And is in the process of committing
•   16/12/20 02:47:49 INFO mapred.LocalJobRunner: 1 / 1 copied.
•   16/12/20 02:47:49 INFO mapred.Task: Task attempt_local1256877917_0001_r_000000_0 is allowed to commit now
•   16/12/20 02:47:49 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1256877917_0001_r_000000_0' to hdfs://Hadoop:54310/user/root/output/_temporary/0/task_local1256877917_0001_r_000000

•   16/12/20 02:47:49 INFO mapred.Task: Task 'attempt_local1256877917_0001_r_000000_0' done.
•   16/12/20 02:47:49 INFO mapred.LocalJobRunner: Finishing task: attempt_local1256877917_0001_r_000000_0
•   16/12/20 02:47:49 INFO mapred.LocalJobRunner: reduce task executor complete.
•   16/12/20 02:47:49 INFO mapreduce.Job:  map 100% reduce 100%
•   16/12/20 02:47:49 INFO mapreduce.Job: Job job_local1256877917_0001 completed successfully
•   16/12/20 02:47:50 INFO mapreduce.Job: Counters: 35

•   16/12/20 02:47:50 INFO streaming.StreamJob: Output directory: /user/root/output

1 个答案:

答案 0 :(得分：0)

根据您的日志

•   16/12/20 02:43:52 INFO mapred.FileInputFormat: Total input paths to process : 1
•   16/12/20 02:43:52 INFO mapreduce.JobSubmitter: number of splits:1

hadoop将整个文件用作单个分割。

尝试使用NLineInputFormat在几台计算机上的映射器之间拆分输入

相关问题
最新问题