显示100%后s3distcp挂起

时间:2014-05-21 20:20:45

标签: hadoop amazon-s3 hdfs amazon-emr

为了尝试使用Amazon EMR解决performance issues,我尝试使用s3distcp将文件从S3复制到我的EMR群集以进行本地处理。作为第一个测试,我使用--groupBy选项从单个目录复制一天的数据,即2160个文件,将它们折叠成一个(或几个)文件。

这项工作似乎运行得很好,向我显示地图/减少进度到100%,但此时进程挂起并且永远不会回来。我怎样才能弄清楚发生了什么?

源文件是存储在S3中的GZip文本文件,每个文件大约30kb。这是一个vanilla Amazon EMR集群,我正在从主节点的shell运行s3distcp。

hadoop@ip-xxx:~$ hadoop jar /home/hadoop/lib/emr-s3distcp-1.0.jar --src s3n://xxx/click/20140520 --dest hdfs:////data/click/20140520 --groupBy ".*(20140520).*" --outputCodec lzo
14/05/21 20:06:32 INFO s3distcp.S3DistCp: Running with args: [Ljava.lang.String;@26f3bbad
14/05/21 20:06:35 INFO s3distcp.S3DistCp: Using output path 'hdfs:/tmp/9f423c59-ec3a-465e-8632-ae449d45411a/output'
14/05/21 20:06:35 INFO s3distcp.S3DistCp: GET http://169.254.169.254/latest/meta-data/placement/availability-zone result: us-west-2b
14/05/21 20:06:35 INFO s3distcp.S3DistCp: Created AmazonS3Client with conf KeyId AKIAJ5KT6QSV666K6KHA
14/05/21 20:06:37 INFO s3distcp.FileInfoListing: Opening new file: hdfs:/tmp/9f423c59-ec3a-465e-8632-ae449d45411a/files/1
14/05/21 20:06:38 INFO s3distcp.S3DistCp: Created 1 files to copy 2160 files
14/05/21 20:06:38 INFO mapred.JobClient: Default number of map tasks: null
14/05/21 20:06:38 INFO mapred.JobClient: Setting default number of map tasks based on cluster size to : 72
14/05/21 20:06:38 INFO mapred.JobClient: Default number of reduce tasks: 3
14/05/21 20:06:39 INFO security.ShellBasedUnixGroupsMapping: add hadoop to shell userGroupsCache
14/05/21 20:06:39 INFO mapred.JobClient: Setting group to hadoop
14/05/21 20:06:39 INFO mapred.FileInputFormat: Total input paths to process : 1
14/05/21 20:06:39 INFO mapred.JobClient: Running job: job_201405211343_0031
14/05/21 20:06:40 INFO mapred.JobClient:  map 0% reduce 0%
14/05/21 20:06:53 INFO mapred.JobClient:  map 1% reduce 0%
14/05/21 20:06:56 INFO mapred.JobClient:  map 4% reduce 0%
14/05/21 20:06:59 INFO mapred.JobClient:  map 36% reduce 0%
14/05/21 20:07:00 INFO mapred.JobClient:  map 44% reduce 0%
14/05/21 20:07:02 INFO mapred.JobClient:  map 54% reduce 0%
14/05/21 20:07:05 INFO mapred.JobClient:  map 86% reduce 0%
14/05/21 20:07:06 INFO mapred.JobClient:  map 94% reduce 0%
14/05/21 20:07:08 INFO mapred.JobClient:  map 100% reduce 10%
14/05/21 20:07:11 INFO mapred.JobClient:  map 100% reduce 19%
14/05/21 20:07:14 INFO mapred.JobClient:  map 100% reduce 27%
14/05/21 20:07:17 INFO mapred.JobClient:  map 100% reduce 29%
14/05/21 20:07:20 INFO mapred.JobClient:  map 100% reduce 100%
[hangs here]

这份工作显示为:

hadoop@xxx:~$ hadoop job -list
1 job currently running
JobId   State   StartTime       UserName        Priority        SchedulingInfo
job_201405211343_0031   1       1400702799339   hadoop  NORMAL  NA

并且目标HDFS目录中没有任何内容:

hadoop@xxx:~$ hadoop dfs -ls /data/click/

有什么想法吗?

2 个答案:

答案 0 :(得分:0)

hadoop @ ip-xxx:〜$ hadoop jar /home/hadoop/lib/emr-s3distcp-1.0.jar --src s3n:// xxx / click / 20140520 ** / ** - hdfs:/ /// data / click / 20140520 ** / ** --groupBy"。(20140520)。" --outputCodec lzo

我遇到了类似的问题。我只需要在目录的末尾添加一个额外的斜杠。因此,它完成了,并且显示了统计数据,上限为100%

答案 1 :(得分:0)

使用s3://而不是s3n。

hadoop jar /home/hadoop/lib/emr-s3distcp-1.0.jar --src s3:// xxx / click / 20140520 --dest hdfs://// data / click / 20140520 --groupBy&# 34; (20140520)&#34。 --outputCodec lzo