Question

我正在尝试将Kafka使用者源与流式文件接收器一起使用到S3

final StreamingFileSink<String> sink = StreamingFileSink
        .forRowFormat(new Path(“s3a://test/test”), new SimpleStringEncoder("UTF-8"))
        .withBucketAssigner(new DateTimeBucketAssigner(outputBucketFormat))
        .withRollingPolicy(rollingPolicy)
        .build();

我有added s3a.access.key和s3a.secret.key到fink-conf.yaml，并按照文档中的说明将./opt/flink-s3-fs-hadoop-1.9.0.jar复制到./plugins/s3-fs-hadoop/

但是它不起作用，它不会在日志中引发任何异常，而只是阻塞处理，如果我使用FileSystem接收器，它会很好地工作。

处理停留在以下日志中：

2020-01-06 23:51:47,237 INFO  org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.FluentPropertyBeanIntrospector - 
Error when creating PropertyDescriptor for public final void org.apache.flink.fs.shaded.hadoop3.org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.

2020-01-06 23:51:47,249 WARN  org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsConfig - 
Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties

2020-01-06 23:51:47,275 INFO  org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsSystemImpl - 
Scheduled Metric snapshot period at 10 second(s).

2020-01-06 23:51:47,276 INFO  org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsSystemImpl - 
s3a-file-system metrics system started

S3是否需要任何附加配置？已针对两个Flink版本：1.9.0和1.9.1进行测试。

找到了https://issues.apache.org/jira/browse/FLINK-14574，但是即使将flink-s3-fs-hadoop jar放在lib下也不起作用。请让我知道docker-hub上是否有任何带有flink映像的工作配置

带有将文件接收器蒸到S3的Flink 1.9不起作用

0 个答案: