使用oozie在S3中移动文件

Question

我想使用AWS oozie在S3中移动文件。我想跑

aws s3 mv s3://temp/*.zip s3://temp/processed_files/. --recursive

我怎么能在oozie中做到这一点？

编辑1

2015-11-12 10:18:55,758  WARN ShellActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] Launcher exception: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:93)
    at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:55)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
    at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:65)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:452)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 17 more

2015-11-12 10:18:55,838  INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] ERROR is considered as FAILED for SLA
2015-11-12 10:18:55,880  INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@killemail] Start action [0000118-151029144311676-oozie-oozi-W@killemail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

Hue Oozie屏幕截图和错误日志的屏幕截图。

Answer 1

如果你做了Helmut Zechmann所说的并且它仍然不能正常工作，请检查问题是否与你的shell脚本没有链接，后者有windows行结尾而不是unix one。

尝试使用notepad ++或其他工具以正确的格式转换它。

Answer 2

编写一个java类来移动s3中的文件，并将其作为oozie中的java动作实现。

Answer 3

问题是oozie无法找到您的脚本。将脚本放入hdfs并使用shell操作的文件元素为脚本提供alies：

$('#id').popover({
    trigger : 'hover',
    delay : {
        show : 1,
        hide : 10000
    },
    html : true,
    content : function() {
        var myhtml = $('#mycontents').html();
        $('#mycontents', myhtml).removeAttr('style');
        return myhtml
    }
})

这将从hdfs路径创建一个别名，该别名是<file>hdfs://path/to/move.sh#move.sh</file> 之后提供的名称，在本例中为move.sh. 现在，您可以在脚本的#元素中引用move.sh。

请参阅https://oozie.apache.org/docs/3.3.0/DG_ShellActionExtension.html以供参考。