Question

Java版本：8 亚军：数据流

在升级到Beam 2.5.0之前，一切正常。例外：

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
    at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
    at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
    at com.geotab.bigdata.streaming.mapserver.backfill.MapServerBatchBeamApplication.main(MapServerBatchBeamApplication.java:115)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
    ... 4 more
Caused by: java.lang.NoSuchMethodError: com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(Ljava/lang/String;)Lcom/google/api/client/googleapis/services/AbstractGoogleClient$Builder;
    at com.google.api.services.storage.Storage$Builder.setBatchPath(Storage.java:9307)
    at com.google.api.services.storage.Storage$Builder.<init>(Storage.java:9286)
    at org.apache.beam.sdk.util.Transport.newStorageClient(Transport.java:95)
    at org.apache.beam.sdk.util.GcsUtil$GcsUtilFactory.create(GcsUtil.java:96)
    at org.apache.beam.sdk.util.GcsUtil$GcsUtilFactory.create(GcsUtil.java:84)
    at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
    at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
    at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)
    at com.sun.proxy.$Proxy25.getGcsUtil(Unknown Source)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:88)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:61)

我的gradle配置：

dependencies {
    compile 'org.apache.beam:beam-runners-direct-java:2.5.0'
    compile 'org.apache.beam:beam-runners-reference-parent:2.4.0'
    compile 'org.apache.beam:beam-runners-reference-java:2.5.0'
    compile ('org.apache.beam:beam-runners-google-cloud-dataflow-java:2.5.0') {
        exclude group: 'com.google.guava'
    }

    compile ('com.google.api-client:google-api-client:1.22.0') {
        force = true //version 1.23.0 causes HTTP response 404 not found exception in Beam 
       //Is this still the case?  Which version should I use with beam 2.5.0?
        exclude group: 'com.google.guava'
    }
    compile 'com.google.guava:guava:23.0'
    compile 'com.typesafe:config:1.3.2'
    compile 'org.apache.httpcomponents:httpclient:4.5.5'
}

侧面问题是：

哪个Google-api-client版本与Beam 2.5兼容？
哪个com.google.guava：guava版本与Beam和google-api-client兼容？

Answer 1

该错误表明AbstractGoogleClient.Builder丢失。检查google-api-java-client版本的文档，该文档存在于1.19和1.20中，但是从1.21开始没有找到。

根据this GitHub issue report，至少更新为1.23可解决这些问题。还要注意Nilson Pontello的这一评论：“从1.22.0更改为1.23.0后，您必须进行干净的构建，否则仍将使用旧的库，尤其是在使用maven生成war文件的情况下。”

Answer 2

用于Java的Cloud Dataflow SDK 2.5.0具有以下编译和runtime dependencies：

+-----------------------+-------------------+----------+
|        GroupId        |     ArtifactId    |  Version |
+-----------------------+-------------------+----------+
| com.google.api-client | google-api-client |   1.23.0 |
| com.google.guava      | guava             |     20.0 |
+-----------------------+-------------------+----------+

Apache Beam：升级到2.5.0后的异常

2 个答案: