GCP数据流 - SSLHandshakeException

时间：2017-07-19 12:27:59

标签： java google-cloud-dataflow

我在运行大型（> 1000 cpus）和中型（100-1000 cpus）数据流作业时遇到以下情况：

  

exception：＆＃34; javax.net.ssl.SSLHandshakeException：远程主机已关闭   握手期间的连接

错误不是致命的，但是，一旦开始它大约每30秒重复一次。显示此错误的作业永远不会完成（我等待超过预期运行时间的4倍）并产生非常有限的结果（少于预期输出的4％）。有限的产量，在生产时，在工作的早期收到，之后不再产生。

我同时使用BigQueryIO和JdbcIO Apache光束源和接收器。

重要的是要注意我的工作在6月初正常工作，但自7月初以来就开始出现这种错误。

我在Google的企业支持中有一个公开案例，但我们只是说结果不会即将发布。从谷歌那里获得的唯一兴趣点是错误可能发生，以及＃34;如果工作人员扩大并大量访问云存储＆＃34;。

但是，该声明没有附加解决方案。 这是日志中记录的完整错误的示例：

   exception:  "javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
    at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call(ResilientOperation.java:166)
    at com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:66)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.getMetadata(GoogleCloudStorageReadChannel.java:573)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStreamAndSetMetadata(GoogleCloudStorageReadChannel.java:645)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:560)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:289)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at java.io.InputStream.read(InputStream.java:101)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:81)
    at org.apache.beam.sdk.util.VarInt.decodeLong(VarInt.java:79)
    at org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:63)
    at org.apache.beam.runners.dataflow.internal.IsmFormat$KeyPrefixCoder.decode(IsmFormat.java:709)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader.readKey(IsmReader.java:1001)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader.access$2000(IsmReader.java:79)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader$WithinShardIsmReaderIterator.advance(IsmReader.java:953)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader$WithinShardIsmReaderIterator.start(IsmReader.java:943)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:581)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:570)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmCacheLoader.call(IsmReader.java:555)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4904)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3628)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2336)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2295)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2208)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
    at com.google.cloud.dataflow.worker.repackaged.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4899)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader.fetch(IsmReader.java:606)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader.getBlock(IsmReader.java:771)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader.access$1000(IsmReader.java:79)
    at com.google.cloud.dataflow.worker.runners.worker.IsmReader$IsmPrefixReaderIterator.get(IsmReader.java:642)
    at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators.getUsingLong(IsmSideInputReader.java:679)
    at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators.access$1300(IsmSideInputReader.java:625)
    at com.google.cloud.dataflow.worker.runners.worker.IsmSideInputReader$ListOverReaderIterators$ListIteratorOverReaderIterators.next(IsmSideInputReader.java:720)
    at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042)
    at com.application.strategy.simulator.MainStrategySimulator$1.processElement(MainStrategySimulator.java:224)
    at com.application.strategy.simulator.MainStrategySimulator$1$auxiliary$4N23tth9.invokeProcessElement(Unknown Source)
    at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:199)
    at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:157)
    at com.google.cloud.dataflow.worker.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:244)
    at com.google.cloud.dataflow.worker.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
    at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:48)
    at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:52)
    at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:198)
    at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
    at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:72)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:336)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.doWork(DataflowWorker.java:295)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:242)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:123)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:103)
    at com.google.cloud.dataflow.worker.runners.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:90)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
    at sun.security.ssl.InputRecord.read(InputRecord.java:505)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
    ... 69 more

1 个答案:

答案 0 :(得分：0)

在线查看此问题的其他实例后，使用Java 7时似乎只会发生错误。这是您的Java版本吗？

如果是，我建议尝试使用Java 8并查看是否可以解决问题。如果我们能够进一步帮助，请告诉我们。

相关的github问题：

https://github.com/spotify/scio/issues/604

https://github.com/ljader/redmine-mylyn-plugin/issues/67

相关问题
最新问题