Question

我正在尝试使用Fargate在ECS上运行Flink 1.7.2。我已经为我的工作设置了状态后端为RocksDB，其路径为= s3：// ...

在我的Dockerfile中，我的基本映像是1.7.2-hadoop27-scala_2.11，我运行以下2条命令：

RUN echo "fs.s3a.aws.credentials.provider: org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider" >> "$FLINK_CONF_DIR/flink-conf.yaml"
RUN cp /opt/flink/opt/flink-s3-fs-hadoop-1.7.2.jar /opt/flink/lib/flink-s3-fs-hadoop-1.7.2.jar

就像它在 https://issues.apache.org/jira/browse/FLINK-8439

但是我得到以下异常：

Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider not found
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:592)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:556)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:256)
        at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:125)
        at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:395)
        at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:318)
        at org.apache.flink.core.fs.Path.getFileSystem(Path.java:298)
        at org.apache.flink.runtime.state.filesystem.FsCheckpointStorage.<init>(FsCheckpointStorage.java:58)
        at org.apache.flink.runtime.state.filesystem.FsStateBackend.createCheckpointStorage(FsStateBackend.java:444)
        at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createCheckpointStorage(RocksDBStateBackend.java:407)
        at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.<init>(CheckpointCoordinator.java:249)
        ... 17 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider not found
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2375)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2446)
        at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:589)
        ... 28 more

在flink-s3-fs-hadoop-1.7.2.jar中，我发现类ContainerCredentialsProvider的包实际上是org.apache.flink.fs.s3base.shaded.com.amazonaws.auth

我已经尝试过：

将aws-sdk-core jar添加到lib中，并将凭据提供者设置为com.amazonaws.auth.ContainerCredentialsProvider（不带阴影），但是我遇到了上面问题链接中提到的问题
将凭据提供者设置为org.apache.flink.fs.s3base.shaded.com.amazonaws.auth.ContainerCredentialsProvider，然后S3FileSystemFactory.java中的代码将其前缀为org.apache.flink.fs.s3hadoop.shaded.

这里有什么关于上课的想法吗？

Answer 1

此问题在以下版本之一中已解决。

我在 Flink 1.9.0 群集上使用以下行运行了该

：

RUN echo "fs.s3a.aws.credentials.provider: com.amazonaws.auth.ContainerCredentialsProvider" >> "$FLINK_CONF_DIR/flink-conf.yaml"

找到该类并开始工作。

您可以在以下位置看到： https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java

FLINK_SHADING_PREFIX现在正确

ECS中的Flink无法找到阴影的ContainerCredentialsProvider

1 个答案: