Question

我正在尝试使用EMR 4.0.0上的步骤运行shell命令，并使用此链接作为参考 - http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html

但我想知道从哪里投入＆＃39; command-runner.jar＆＃39;在JAR地点＆＃39;字段 http://i.stack.imgur.com/CRicz.png

我保留了＆＃39; command-runner.jar＆＃39;在AWS s3中尝试从该位置加载它并在＆＃39; Arguments＆＃39;给了我的＆＃39; example.sh＆＃39;的s3位置。文件和添加步骤后，它无法提供此异常

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:139)
    at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:92)
    ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 8 more

Answer 1

要运行存储在S3中的shell脚本，请使用位于script-runner.jar的{{1}}。

s3://elasticmapreduce/libs/script-runner/script-runner.jar仅适用于本地文件。

AWS EMR 4.0 - 如何添加自定义JAR步骤以运行shell命令

1 个答案: