Question

我试图通过（现有的）AppEngine应用程序来启动数据流作业。 DataFlow作业读取存储在DataStore中的GAE应用程序生成的数据，并将处理后的数据写入BigQuery。我收到了以下错误。

java.lang.SecurityException: Google App Engine does not support Runtime.addShutdownHook
at com.google.appengine.runtime.Request.process-a010d936cef53bc8(Request.java)
at java.lang.Runtime.addShutdownHook(Runtime.java:46)
at com.google.cloud.dataflow.sdk.repackaged.com.google.common.util.concurrent.MoreExecutors$Application.addShutdownHook(MoreExecutors.java:232)
at com.google.cloud.dataflow.sdk.repackaged.com.google.common.util.concurrent.MoreExecutors$Application.addDelayedShutdownHook(MoreExecutors.java:204)
at com.google.cloud.dataflow.sdk.repackaged.com.google.common.util.concurrent.MoreExecutors$Application.getExitingExecutorService(MoreExecutors.java:188)
at com.google.cloud.dataflow.sdk.repackaged.com.google.common.util.concurrent.MoreExecutors.getExitingExecutorService(MoreExecutors.java:89)
at com.google.cloud.dataflow.sdk.util.BigQueryTableInserter.<clinit>(BigQueryTableInserter.java:79)
at com.google.cloud.dataflow.sdk.io.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:886)
at com.google.cloud.dataflow.sdk.io.BigQueryIO$Write$Bound.apply(BigQueryIO.java:942)
at com.google.cloud.dataflow.sdk.io.BigQueryIO$Write$Bound.apply(BigQueryIO.java:724)
at com.google.cloud.dataflow.sdk.runners.PipelineRunner.apply(PipelineRunner.java:74)
at com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner.apply(DataflowPipelineRunner.java:327)
at com.google.cloud.dataflow.sdk.Pipeline.applyInternal(Pipeline.java:367)
at com.google.cloud.dataflow.sdk.Pipeline.applyTransform(Pipeline.java:274)
at com.google.cloud.dataflow.sdk.values.PCollection.apply(PCollection.java:161)

有没有办法在通过GAE启动时在数据流中写入BigQuery表？我将跑步者设置为DataflowPipelineRunner，因此它不应该尝试在GAE实例上运行它。（有没有办法验证？）

Answer 1

DataflowPipelineRunner正在尝试在将作业提交到Google Cloud Dataflow服务之前验证您的管道。在此堆栈跟踪中，我们尝试验证目标BigQuery表是否为空。

在此过程中，我们会初始化ExecutorService，但不允许在Google App Engine中运行。这是不幸的，因为在这种情况下并不严格需要。此修复程序被跟踪为BEAM-142，请检查是否有任何更新。

解决方法是在App Engine环境中禁用验证。在BigQuery接收器中使用BigQueryIO.Write.withoutValidation()。

Answer 2

您可以尝试App Engine灵活环境，而不是像允许的JRE类那样限制App Engine Standard。

使用针对BigQueryTableInserter的addShutdownHook上的SecurityException启动带有App Engine错误的Dataflow作业

2 个答案: