Question

我正在尝试在EMR集群上运行distcp命令：

hadoop distcp s3a://... hdfs://host/data/...

当我运行它时，它会出现以下错误：

Exit code: 1
Exception message: /bin/bash: /mnt/yarn/logs/application_1524773139099_0003/container_1524773139099_0003_02_000001/stdout: No such file or directory

Stack trace: ExitCodeException exitCode=1: /bin/bash: /mnt/yarn/logs/application_1524773139099_0003/container_1524773139099_0003_02_000001/stdout: No such file or directory

at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
at org.apache.hadoop.util.Shell.run(Shell.java:479)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

我已经检查了集群中的所有节点，并且它们都有我创建的/mnt/yarn/logs目录。这是怎么回事？

Answer 1

确保运行该作业的用户具有足够的权限，可以在路径/ mnt / yarn / logs中创建临时目录，例如application _ *******。最好使用配置单元用户并为application_1524773139099_0003拉出纱线日志，以查看可能解释实际错误的错误。

由于缺少纱线日志目录，hadoop distcp失败

1 个答案: