Question

我正在尝试构建一个从BigQuery读取的Dataflow作业，并将消息发送到Pubsub主题。我一直在努力解决相互依赖问题，这是最新的问题。

数据流作业启动正常，但失败并出现以下异常：

com.google.cloud.pubsub.PubSubException：io.grpc.StatusRuntimeException：位于com.google.cloud.pubsub上的com.google.cloud.pubsub.spi.DefaultPubSubRpc $ 1.apply（DefaultPubSubRpc.java:174）的UNAVAILABLE .spi.DefaultPubSubRpc $ 1.apply（DefaultPubSubRpc.java:168）com.google.common.util.concurrent.Futures $ CatchingFuture.doFallback（Futures.java:842）at com.google.common.util.concurrent.Futures $ Cat.Future.doFallback（Futures.java:834）位于com.google.common.util.concurrent.Futures $ AbstractCatchingFuture.run（Futures.java:789）com.google.common.util.concurrent.MoreExecutors $ DirectExecutor.execute（更多EXecutors.java:456）com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）at com.google.common.util.concurrent.AbstractFuture.complete（AbstractFuture.java:753）at com位于com.google.api.gax的com.google.common.util.concurrent.SettableFuture.setException（SettableFuture.java:53）上的.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634）。来自com.google.common.util.concurrent.MoreExecutors的com.google.common.util.concurrent.Futures $ 6.run（Futures.java:1764）中的grpc.RetryingCallable $ Retryer $ 1.onFailure（RetryingCallable.java:139） com.google.common.util.concurrent.AbstractFuture.complete（AbstractFuture.java）上的com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）中的DirectExecutor.execute（MoreExecutors.java:456）： 753）com.google.com上的com.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634）com.google.api上的com.google.common.util.concurrent.SettableFuture.setException（SettableFuture.java:53）位于com.google.common.util.concurrent.MoreExecutors的com.google.common.util.concurrent.Futures $ 6.run（Futures.java:1764）上的.gax.grpc.ExceptionTransformingCallable $ 1.onFailure（ExceptionTransformingCallable.java:91）来自com.google.common.util.concurrent的com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）上的$ DirectExecutor.execute（MoreExecutors.java:456） .AbstractFuture.complete（AbstractFuture.java:753）at com.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634）at io.grpc.stub.ClientCalls $ GrpcFuture.setException（ClientCalls.java:382））在io.grpc.internal.ContextRunnable.run的io.grpc.inub.ClientCallImpl $ ClientStreamListenerImpl $ 3.runInContext（ClientCallImpl.java:462）的io.grpc.stub.ClientCalls $ UnaryStreamToFuture.onClose（ClientCalls.java:358）（ContextRunnable.java:54）位于java.util的java.util.concurrent.Executors $ RunnableAdapter.call（Executors.java:511）的io.grpc.internal.SerializingExecutor $ TaskRunner.run（SerializingExecutor.java:154）。 concurrent.FutureTask.run（FutureTask.java:266）at java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 201（ScheduledThreadPoolExecutor.java:180）at java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor.java:293））引起：com.google.api.gax.grpc.ApiException：io.grpc.StatusRuntimeException：UNAVAIL ABLE at com.google.api.gax.grpc.ExceptionTransformingCallable $ 1.onFailure（ExceptionTransformingCallable.java:91）com.google.com上的com.google.common.util.concurrent.Futures $ 6.run（Futures.java:1764）。 common.util.concurrent.MoreExecutors com.google.common.util.concurrent上的com.google.common.util.concurrent.AbstractFuture.executeListener（AbstractFuture.java:817）中的$ DirectExecutor.execute（MoreExecutors.java:456）。来自com.google.common.util.concurrent.AbstractFuture.setException（AbstractFuture.java:634）的AbstractFuture.complete（AbstractFuture.java:753）at io.grpc.stub.ClientCalls $ GrpcFuture.setException（ClientCalls.java:382） at io.grpc.stub.ClientCalls $ UnaryStreamToFuture.onClose（ClientCalls.java:358）at io.grpc.internal.ClientCallImpl $ ClientStreamListenerImpl $ 3.runInContext（ClientCallImpl.java:462）at io.grpc.internal.ContextRunnable.run（ ContextRunnable.java:54）at io.grpc.internal.SerializingExecutor $ TaskRunner.run（SerializingExecutor.java:154）at java.util.concurrent.Executors $ RunnableAdapte r.call（Executors.java:511）在Java.util.concurrent.FutureTask.run（FutureTask.java:266）java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 201（ScheduledThreadPoolExecutor.java:180）在java。 util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor.java:293）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java： 617）at java.lang.Thread.run（Thread.java:745）引起：io.grpc.StatusRuntimeException：UNAVAILABLE at io.grpc.Status.asRuntimeException（Status.java:431）... 11更多引起： javax.net.ssl.SSLHandshakeException：在sun.security.ssl.SSLEngineImpl.kickstartHandshake（SSLEngineImpl）的sun.security.ssl.Handshaker.activate（Handshaker.java:503）中没有合适的协议（协议被禁用或密码套件不合适） .java：729）在io.net的sun.security.ssl.SSLEngineImpl.beginHandshake（SSLEngineImpl.java:756） ty.handler.ssl.JdkSslEngine.beginHandshake（JdkSslEngine.java:147）在io.netty.handler.ssl.SslHandler.handshake（SslHandler.java:1363）在io.netty.handler.ssl.SslHandler.channelActive（SslHandler。 java：1405）at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelActiveNow（ChannelHandlerInvokerUtil.java:48）at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelActive（DefaultChannelHandlerInvoker.java:79）at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive（AbstractChannelHandlerContext的.java：126）在io.netty.channel.DefaultChannelPipeline.fireChannelActive（DefaultChannelPipeline.java:884）在io.netty.channel.nio.AbstractNioChannel $ AbstractNioUnsafe.fulfillConnectPromise（AbstractNioChannel.java:260）在io.netty.channel。 nio.AbstractNioChannel $ AbstractNioUnsafe.finishConnect（AbstractNioChannel.java:290）at io.netty.channel.nio.NioEventLoop.processSelectedKey（NioEventLoop.java:527）at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized（NioEventLoop.java： 467）在io.netty.channel.nio.NioEventLoop.processSelectedKeys（NioEventLoop.java:381）在io.netty.channel.nio.NioEventLoop.run（NioEventLoop.java:353）在io.netty.util.concurrent.SingleThreadEventExecutor $ 5 run（SingleThreadEventExecutor.java:742）... 3 more

我怀疑它与正在使用的netty / ssl库有关，但我不知道正确的行动方案是什么。我花了很多时间尝试从生成的阴影jar中排除某些库的某些版本，其中类重叠，但结果是找不到类的异常。

这些是我的依赖项：

<dependencies>
  <dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>1.8.0</version>
  </dependency>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-pubsub</artifactId>
    <version>0.3.0</version>
  </dependency>
  <dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>3.0.0-beta-1</version>
  </dependency>
</dependencies>

这是依赖树：

[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ dataflow-bulk-load ---
[INFO] com.google.lindsaysmith.titan:dataflow-bulk-load:jar:0.0.1-SNAPSHOT
[INFO] +- com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:1.8.0:compile
[INFO] |  +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev36-1.22.0:compile
[INFO] |  +- io.grpc:grpc-all:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-auth:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-netty:jar:0.13.1:compile
[INFO] |  |  |  \- io.netty:netty-codec-http2:jar:4.1.0.CR1:compile
[INFO] |  |  |     \- io.netty:netty-codec-http:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-core:jar:0.13.1:compile
[INFO] |  |  +- io.grpc:grpc-okhttp:jar:0.13.1:compile
[INFO] |  |  |  +- com.squareup.okio:okio:jar:1.6.0:compile
[INFO] |  |  |  \- com.squareup.okhttp:okhttp:jar:2.5.0:compile
[INFO] |  |  +- io.grpc:grpc-protobuf-nano:jar:0.13.1:compile
[INFO] |  |  |  \- com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5:compile
[INFO] |  |  \- io.grpc:grpc-stub:jar:0.13.1:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.netty:netty-buffer:jar:4.1.0.CR1:compile
[INFO] |  |  |  \- io.netty:netty-common:jar:4.1.0.CR1:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.0.CR1:compile
[INFO] |  |  |  \- io.netty:netty-resolver:jar:4.1.0.CR1:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.1.0.CR1:compile
[INFO] |  +- com.google.api.grpc:grpc-pubsub-v1:jar:0.0.2:compile
[INFO] |  |  \- com.google.api.grpc:grpc-core-proto:jar:0.0.3:compile
[INFO] |  +- com.google.auth:google-auth-library-oauth2-http:jar:0.4.0:compile
[INFO] |  |  \- com.google.auth:google-auth-library-credentials:jar:0.4.0:compile
[INFO] |  +- com.google.cloud.bigtable:bigtable-protos:jar:0.3.0:compile
[INFO] |  |  \- com.google.guava:guava:jar:19.0:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0:compile
[INFO] |  +- com.google.http-client:google-http-client:jar:1.22.0:compile
[INFO] |  |  \- org.apache.httpcomponents:httpclient:jar:4.0.1:compile
[INFO] |  |     +- org.apache.httpcomponents:httpcore:jar:4.0.1:compile
[INFO] |  |     +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |  |     \- commons-codec:commons-codec:jar:1.3:compile
[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client-java6:jar:1.22.0:compile
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.22.0:compile
[INFO] |  +- com.google.apis:google-api-services-datastore-protobuf:jar:v1beta2-rev1-4.0.0:compile
[INFO] |  +- com.google.cloud.bigdataoss:gcsio:jar:1.4.5:compile
[INFO] |  |  +- com.google.api-client:google-api-client-java6:jar:1.22.0:compile
[INFO] |  |  \- com.google.api-client:google-api-client-jackson2:jar:1.22.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:1.1.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-protobuf:jar:1.22.0:compile
[INFO] |  |  \- com.google.http-client:google-http-client-jackson:jar:1.22.0:compile
[INFO] |  +- com.google.cloud.datastore:datastore-v1-protos:jar:1.0.1:compile
[INFO] |  +- com.google.cloud.bigdataoss:util:jar:1.4.5:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.7.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.0:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.14:compile
[INFO] |  +- org.apache.avro:avro:jar:1.7.7:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.2.1:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.9:compile
[INFO] |  \- joda-time:joda-time:jar:2.4:compile
[INFO] +- com.google.cloud:google-cloud-pubsub:jar:0.3.0:compile
[INFO] |  +- io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork17:compile
[INFO] |  +- com.google.cloud:google-cloud-core:jar:0.3.0:compile
[INFO] |  |  +- com.google.api-client:google-api-client-appengine:jar:1.21.0:compile
[INFO] |  |  |  +- com.google.oauth-client:google-oauth-client-appengine:jar:1.21.0:compile
[INFO] |  |  |  |  +- com.google.oauth-client:google-oauth-client-servlet:jar:1.21.0:compile
[INFO] |  |  |  |  |  \- com.google.http-client:google-http-client-jdo:jar:1.21.0:compile
[INFO] |  |  |  |  \- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  |  |  +- com.google.api-client:google-api-client-servlet:jar:1.21.0:compile
[INFO] |  |  |  |  \- javax.jdo:jdo2-api:jar:2.3-eb:compile
[INFO] |  |  |  |     \- javax.transaction:transaction-api:jar:1.1:compile
[INFO] |  |  |  \- com.google.http-client:google-http-client-appengine:jar:1.21.0:compile
[INFO] |  |  +- org.json:json:jar:20151123:compile
[INFO] |  |  +- com.google.api:gax:jar:0.0.16:compile
[INFO] |  |  |  \- com.google.inject:guice:jar:4.0:compile
[INFO] |  |  |     +- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |     \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  \- com.google.api.grpc:grpc-google-common-protos:jar:0.0.7:compile
[INFO] |  +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.0.7:compile
[INFO] |  \- com.google.auto.value:auto-value:jar:1.1:compile
[INFO] \- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.749s
[INFO] Finished at: Sun Dec 04 16:50:41 CET 2016
[INFO] Final Memory: 16M/972M
[INFO] ------------------------------------------------------------------------

Answer 1

当我尝试使用PubSub库为DataFlow中的每个已处理项目手动发送消息时，就会出现此问题。

我没有这样做，而是转而使用DataFlow的PubSub IO Sink类发送消息，这完美无缺。

文档：

https://cloud.google.com/dataflow/model/pubsub-io#writing-with-pubsubio

数据流中的错误：io.grpc.StatusRuntimeException：UNAVAILABLE

1 个答案: