在读取和写入BigQuery期间，Dataflow SDK 2.2.0异常

时间：2018-02-22 07:43:32

标签： google-cloud-dataflow apache-beam

我正在迁移到Dataflow SDK 2.2.0。 我按照这个例子创建示例代码： https://beam.apache.org/get-started/wordcount-example/ 

PCollection<TableRow> rows = p.apply(BigQueryIO.readTableRows()
    .from(BigQueryHelpers.parseTableSpec("my-project:dataflow_test.test")));

rows.apply("Write", BigQueryIO.writeTableRows()
    .to(BigQueryHelpers.parseTableSpec("my-project:dataflow_test.test_output"))
    .withSchema(schema)
    .withCreateDisposition((BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED))
    .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));

我得到了这个例外：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/common/runner/v1/RunnerApi$Components
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.SdkComponents.<init>(SdkComponents.java:58)
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.SdkComponents.create(SdkComponents.java:54)
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation.toProto(WindowingStrategyTranslation.java:284)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.serializeWindowingStrategy(DataflowPipelineTranslator.java:129)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1500(DataflowPipelineTranslator.java:114)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$5.groupByKeyHelper(DataflowPipelineTranslator.java:806)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$5.translate(DataflowPipelineTranslator.java:784)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$5.translate(DataflowPipelineTranslator.java:781)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:442)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:663)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:446)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:386)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:173)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:537)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:170)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at in.cli.apps.acsi.loader.dataflow.SimpleDataflow.runDataflow(SimpleDataflow.java:91)
at in.cli.apps.acsi.loader.dataflow.SimpleDataflow.runDataflow(SimpleDataflow.java:58)
at in.cli.apps.acsi.loader.dataflow.SimpleDataflow.main(SimpleDataflow.java:38)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.common.runner.v1.RunnerApi$Components
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 28 more

我发现了一篇建议添加依赖关系的帖子：

java.lang.NoClassDefFoundError: org/apache/beam/sdk/runners/PipelineRunner after migrating to dataflow 2.x

但是当我将它添加到我的pom.xml中时，我得到另一个错误

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/options/ExperimentalOptions
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.beam.runners.dataflow.DataflowPipelineRegistrar$Options.getPipelineOptions(DataflowPipelineRegistrar.java:42)
at org.apache.beam.sdk.options.PipelineOptionsFactory.initializeRegistry(PipelineOptionsFactory.java:575)
at org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:525)
at com.insightos.utils.dataflow.DataflowHelper.getDataflowOptions(DataflowHelper.java:103)
at in.cli.apps.acsi.loader.dataflow.SimpleDataflow.runDataflow(SimpleDataflow.java:43)
at in.cli.apps.acsi.loader.dataflow.SimpleDataflow.main(SimpleDataflow.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.options.ExperimentalOptions
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 30 more

我被卡在那里 - 无法找到任何有关它的信息。 如何解决这个问题呢？

0 个答案:

没有答案
相关问题
最新问题