Question

我正在尝试将一些文件从S3存储桶复制到我的EMR集群的HDFS。但我收到以下错误：

Exception in thread "main" java.lang.RuntimeException: Error running job
    at com.amazon.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:771)
    at com.amazon.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:580)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at com.amazon.elasticmapreduce.s3distcp.Main.main(Main.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.87.26.26:9000/tmp/33e4f3b9-d29a-49e8-9706-ea70e07e3ff2/files
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:285)
    at org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
    at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
    at com.amazon.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:751)
    ... 9 more

我使用的命令是：

./elastic-mapreduce --jobflow  j-12345678 --jar /home/hadoop/lib/emr-s3distcp-1.0.jar --args '--src,s3n://my-bucket/data/,--dest,hdfs:///data/in,--srcPattern,xyz01-1-1*ped*' --step-name "Copy input files to HDFS" --wait-for-steps

我尝试运行示例字数计数作业，检查HDFS是否有任何问题，但运行正常。

任何人都可以帮我这个吗？如果需要更多信息，请告诉我，我将更新说明。

Answer 1

通常是--srcPattern '<regex>'参数。您还可以使用hadoop fs -cp s3://src/file1.something /my/output/path/测试1个文件并修改正则表达式。同样以.*任何char-0或更多次开头，应放宽匹配。

很高兴知道是否记录正则表达式不匹配以及在哪里。

从S3复制到HDFS时出错

1 个答案: