无法使用Apache Beam / Apache Flink创建父目录

时间:2018-06-15 13:23:13

标签: apache-flink apache-beam

当我尝试使用Apache Beam和Apache Flink使用Docker容器(集群模式)部署测试项目时,出现此错误:

org.apache.beam.sdk.util.UserCodeException: java.io.IOException: Unable to create parent directories for '/home/beam-test/.temp-beam-2018-06-15_12-57-41-0/549f66ca-c1f8-42d7-91e8-dec27f0689b7'
    at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
    at org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
    at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:65)
    at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.mapPartition(FlinkDoFnFunction.java:120)
    at org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
    at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503)
    at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to create parent directories for '/home/beam-test/.temp-beam-2018-06-15_12-57-41-0/549f66ca-c1f8-42d7-91e8-dec27f0689b7'
    at org.apache.beam.sdk.io.LocalFileSystem.create(LocalFileSystem.java:103)
    at org.apache.beam.sdk.io.LocalFileSystem.create(LocalFileSystem.java:78)
    at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:248)
    at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:235)
    at org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
    at org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)

我尝试安装一个docker卷来放置输入和输出文件,但是在尝试创建临时目录的过程中它失败了。

我检查了所有者和组对docker卷的访问权限并且没关系(手动我可以创建和删除里面的文件和目录)。

你知道为什么会发生这种情况吗?

JDK和Flink Docker Image会出现问题吗?

Docker Image:Flink(https://hub.docker.com/r/library/flink/) JDK安装: Openjdk版“1.8.0_171” OpenJDK运行时环境(build 1.8.0_171-8u171-b11-1~deb9u1-b11) OpenJDK 64位服务器VM(版本25.171-b11,混合模式)

0 个答案:

没有答案