Question

我在下面的beamSql程序中指定了dataflow runner：

DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
    options.setStagingLocation("gs://gcpbucket/staging");
    options.setTempLocation("gs://gcpbucket/tmp");
    options.setProject("beta-19xxxx");
    options.setRunner(DataflowRunner.class);
    Pipeline p = Pipeline.create(options);

但我已经超越了例外：

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at my.proj.StarterPipeline.main(StarterPipeline.java:34)Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more Caused by: java.lang.IllegalArgumentException: Unable to use ClassLoader to detect classpath elements. Current ClassLoader is jdk.internal.loader.ClassLoaders$AppClassLoader@782830e, only URLClassLoaders are supported.
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.PipelineResources.detectClassPathResourcesToStage(PipelineResources.java:43)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:262)

任何人都可以帮我理解究竟是什么问题吗？

Answer 1

将java 9降级为java 8就可以了。

Answer 2

对我来说，解决方法有所不同。在我的pom.xml中，以下依赖项设置为2.20，而Apache Beam设置为2.19

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
  <version>2.20</version>
</dependency>

将其更改为与Beam版本相同后，构建工作正常。

Net Net，只需确保将所有与梁相关的SDK设置为相同版本即可。对变量使用相同的变量，例如$ {beamVersion}，这样就不会遇到奇怪的错误。

Answer 3

将Beam SDK与JVM> 8一起使用时可能会发生此问题。

对JVM 11（或9）的支持仅是实验性的，从Beam SDK 2.12开始，至少直到Beam SDK 2.16。这个答案提供了更多背景信息：https://stackoverflow.com/a/57710742/1046584

关于此特定的堆栈跟踪，似乎与以下问题有关：https://issues.apache.org/jira/browse/BEAM-3718

无法从beamSql，apache beam中的工厂方法DataflowRunner＃fromOptions构造实例

3 个答案: