当我尝试使用Apache Beam和Apache Flink使用Docker容器(集群模式)部署测试项目时,出现此错误:
org.apache.beam.sdk.util.UserCodeException: java.io.IOException: Unable to create parent directories for '/home/beam-test/.temp-beam-2018-06-15_12-57-41-0/549f66ca-c1f8-42d7-91e8-dec27f0689b7'
at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.processElement(DoFnRunnerWithMetricsUpdate.java:65)
at org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.mapPartition(FlinkDoFnFunction.java:120)
at org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503)
at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:703)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to create parent directories for '/home/beam-test/.temp-beam-2018-06-15_12-57-41-0/549f66ca-c1f8-42d7-91e8-dec27f0689b7'
at org.apache.beam.sdk.io.LocalFileSystem.create(LocalFileSystem.java:103)
at org.apache.beam.sdk.io.LocalFileSystem.create(LocalFileSystem.java:78)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:248)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:235)
at org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
我尝试安装一个docker卷来放置输入和输出文件,但是在尝试创建临时目录的过程中它失败了。
我检查了所有者和组对docker卷的访问权限并且没关系(手动我可以创建和删除里面的文件和目录)。
你知道为什么会发生这种情况吗?
JDK和Flink Docker Image会出现问题吗?
Docker Image:Flink(https://hub.docker.com/r/library/flink/) JDK安装: Openjdk版“1.8.0_171” OpenJDK运行时环境(build 1.8.0_171-8u171-b11-1~deb9u1-b11) OpenJDK 64位服务器VM(版本25.171-b11,混合模式)