Question

我尝试从我的Flink应用程序从S3n迁移到S3a。我编写了自定义滚动接收器，用于将数据写入aws S3。它接受URL作为基本路径，如S3a：///。它适用于S3n，当我跟踪S3a的更改时，它会抛出异常。看起来core-site.xml似乎没有为AWS凭证获取S3a。它使用AWS默认加载程序来加载凭据。请告诉我这个配置缺失了。现在我在eclipse开发环境中运行，它在flink集群模式下是如何工作的？

例外是：

com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
                at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
                at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
                at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
                at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
                at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2316)
                at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
                at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
                at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
                at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
                at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
                at com.abc.abc.common.sinks.CustomRollingSink.open(CustomRollingSink.java:260)

我正在使用flink 1.0.0版本并依赖于项目pom文件和核心站点.xml以获取AWS凭证。

<property>
<name>fs.s3a.awsAccessKeyId</name>  
<value>value..</value>
</property>
<property>
<name>fs.s3a.awsSecretAccessKey</name>
<value>values...</value>
</property>
<property>
    <name>fs.s3a.impl</name>        
    <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>

Pom文件：

 <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-aws</artifactId>
            <version>2.7.2</version>
        </dependency>
         <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.2.5</version>
        </dependency> 
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.2.5</version>
        </dependency>   
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.6.3</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>2.6.3</version>
        </dependency>

由于未在任何加载程序中找到AWS凭证，因此写入S3a文件系统的Flink应用程序失败

0 个答案: