数据流错误:拒绝拆分

时间:2018-04-03 10:00:45

标签: google-cloud-dataflow

我在运行数据流管道时多次收到以下错误日志:

 $('<div id="calendar" class="col-lg-2 col-md-2 col-xs-2 no-padding-returnFlight"><b>+ Add Return Flight</b></div>').prependTo('.examplefooter-totalFare_wrap');

这项工作最终失败了。 JobId:2018-04-02_00_19_15-14115706867296503746

我正在使用Apache Beam 2.2.0的java实现。它使用带有空键的重新洗牌作为避免融合的中间步骤。它似乎在那一步失败了。

最后一个错误日志是:

Refusing to split <at position ShufflePosition(base64:AAAAAtxW0XoAAQ) of shuffle range [ShufflePosition(base64:AAAAAgD_AP8A_wD_AAE), ShufflePosition(base64:AAAAAtxW0XsAAQ))> at ShufflePosition(base64:AAAAAtxW0XsAAQ): proposed split position out of range

高级别的数据流管道是:

Workflow failed. Causes: S83:Reshuffle2/GroupByKey/Read+Reshuffle2/GroupByKey/GroupByWindow+Reshuffle2/ExpandIterable+Drop key2/Values/Map+WriteTrackerTableToBQ/PrepareWrite/ParDo(Anonymous)+WriteTrackerTableToBQ/BatchLoads/rewindowIntoGlobal/Window.Assign+WriteTrackerTableToBQ/BatchLoads/WriteBundlesToFiles+WriteTrackerTableToBQ/BatchLoads/ReifyResults/View.AsIterable/View.CreatePCollectionView/ParDo(ToIsmRecordForGlobalWindow)+WriteTrackerTableToBQ/BatchLoads/GroupByDestination/Reify+WriteTrackerTableToBQ/BatchLoads/GroupByDestination/Write failed., A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. 

1 个答案:

答案 0 :(得分:0)

&#34;拒绝分裂&#34;错误不应导致作业失败。根据您发布的工作错误消息,我认为我们已经在生产中解决了这个问题。您是否可以尝试再次启动工作,看看是否仍存在相同的错误?