Question

我一直在使用Beam管道examples作为指导，以尝试从S3为管道加载文件。像在示例中一样，我已经定义了自己的PipelineOptions，它也扩展了S3Options，并且尝试使用DefaultAWSCredentialsProviderChain。配置它的代码是：

MyPipelineOptions options = PipelineOptionsFactory.fromArgs(args).as(MyPipelineOptions.class);

options.setAwsCredentialsProvider(new DefaultAWSCredentialsProviderChain());
options.setAwsRegion("us-east-1");

runPipeline(options);

当我从Intellij运行它时，使用Direct Runner可以正常工作但是当我将其打包为jar并执行（也使用Direct Runner）时，我会看到：

Exception in thread "main" java.lang.IllegalArgumentException: PipelineOptions specified failed to serialize to JSON.
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:166)
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
    at a.b.c.beam.CleanSkeleton.runPipeline(CleanSkeleton.java:69)
    at a.b.c.beam.CleanSkeleton.main(CleanSkeleton.java:53)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected IOException (of type java.io.IOException): Failed to serialize and deserialize property 'awsCredentialsProvider' with value 'com.amazonaws.auth.DefaultAWSCredentialsProviderChain@40f33492'
    at com.fasterxml.jackson.databind.JsonMappingException.fromUnexpectedIOE(JsonMappingException.java:338)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3247)
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:163)
    ... 5 more

我正在使用gradle完成以下任务来构建我的jar：

jar {
    manifest {
        attributes (
                'Main-Class': 'a.b.c.beam.CleanSkeleton'
        )
    }
    from {
        configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
    }
    from('src') {
        include '/main/resources/*'
    }



    zip64 true
    exclude 'META-INF/*.RSA', 'META-INF/*.SF', 'META-INF/*.DSA'
}

Answer 1

之所以出现此问题，是因为创建fat / uber jar时，META-INF/serivces中的文件被重复文件覆盖。特别是com.fasterxml.jackson.databind.Module，其中需要定义许多Jackson模块，但缺少这些模块。其中包括org.apache.beam.sdk.io.aws.options.AwsModule和com.fasterxml.jackson.datatype.joda.JodaModule。 DirectRunner中的代码像这样实例化ObjectMapper：

new ObjectMapper()
      .registerModules(ObjectMapper.findModules(ReflectHelpers.findClassLoader()));

ObjectMapper::findModules依靠java.util.ServiceLoader来定位来自META-INF/services/文件的服务。

解决方案是使用gradle Shadow plugin来构建fat / uber jar并将其配置为合并服务文件：

apply plugin: 'com.github.johnrengelman.shadow'

shadowJar {
    mergeServiceFiles()
    zip64 true
}

Beam：无法序列化和反序列化属性'awsCredentialsProvider

1 个答案: