上下文:使用Windows和GroupByKey阶段的数据流2.6.0的流管道
发生了什么:短时间后,管道不断发出以下错误:
java.lang.RuntimeException: Invalid namespace string: '//'
看堆栈跟踪,发生这种情况是因为StateNamespaces
(https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StateNamespaces.java)试图向CoderUtils.decodeFromBase64
发送一个空的命名空间字符串,从而创建了一个未捕获的EOF error
,它本身启动CoderException
。
但是,这仅在云中发生。确实,查看堆栈跟踪记录可以发现,数据流专有代码在这里很危急。 (特别是直接跑步者从不使用StateNamespaces
)
这是堆栈跟踪:(完整堆栈跟踪在此处https://pastebin.com/xwh82pYx)
java.lang.RuntimeException: Invalid namespace string: '//'
org.apache.beam.runners.core.StateNamespaces.fromString(StateNamespaces.java:270)
com.google.cloud.dataflow.worker.WindmillTimerInternals.windmillTimerToTimerData(WindmillTimerInternals.java:264)
com.google.cloud.dataflow.worker.StreamingModeExecutionContext$StepContext.lambda$getNextFiredTimer$1(StreamingModeExecutionContext.java:535)
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.Iterators$7.transform(Iterators.java:750)
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
com.google.cloud.dataflow.worker.StreamingModeExecutionContext$StepContext.getNextFiredTimer(StreamingModeExecutionContext.java:543)
com.google.cloud.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:445)
com.google.cloud.dataflow.worker.SimpleParDoFn.processTimers(SimpleParDoFn.java:343)
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:51)
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:83)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1227)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:136)
com.google.cloud.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:966)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.coders.CoderException: java.io.EOFException
org.apache.beam.sdk.coders.InstantCoder.decode(InstantCoder.java:70)
org.apache.beam.sdk.coders.InstantCoder.decode(InstantCoder.java:34)
...
如何解决此错误?
一些调查: