使用PubSub在本地运行Java数据流管道

时间:2019-12-04 16:46:04

标签: java google-cloud-dataflow apache-beam

我试图在Eclipse中使用Apache Beam在Java数据流中本地运行(使用本地运行器)。

将pubsub添加为数据源时会发生问题。
(如果删除该行并尝试使用简单的管道,则可以毫无问题地执行它)

管道代码:

public class StarterPipeline {

 public static void main(String[] args) {
    Options options = PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
    run(options);
  }

public static PipelineResult run(Options options) {

    Pipeline pipeline = Pipeline.create(options);

    /*
     * Step 1: Read from PubSub
     */
    PCollection<PubsubMessage> messages = null;
    if (options.getUseSubscription()) {
        messages = pipeline.apply("ReadPubSubSubscription", PubsubIO.readMessagesWithAttributes()
                .fromSubscription(options.getInputSubscription()).withIdAttribute("messageId"));
    } else {
        messages = pipeline.apply("ReadPubSubTopic", PubsubIO.readMessagesWithAttributes()
                .fromTopic(options.getInputTopic()).withIdAttribute("messageId"));
    }

     <...>
    return pipeline.run();
}

和错误消息:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/util/DoFnAndMainOutput
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation.translateDoFn(ParDoTranslation.java:462)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation$1.translateDoFn(ParDoTranslation.java:160)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation.payloadForParDoLike(ParDoTranslation.java:695)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation.translateParDo(ParDoTranslation.java:156)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation$ParDoPayloadTranslator.translate(ParDoTranslation.java:111)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.PTransformTranslation.toProto(PTransformTranslation.java:206)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation.getParDoPayload(ParDoTranslation.java:547)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.ParDoTranslation.isSplittable(ParDoTranslation.java:557)
    at org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.PTransformMatchers$4.matches(PTransformMatchers.java:194)
    at org.apache.beam.sdk.Pipeline$2.visitPrimitiveTransform(Pipeline.java:282)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
    at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
    at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
    at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460)
    at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:260)
    at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:210)
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:170)
    at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:66)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:315)
    at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
    at com.accenture.pipeline.StarterPipeline.run(StarterPipeline.java:128)
    at com.accenture.pipeline.StarterPipeline.main(StarterPipeline.java:74)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.util.DoFnAndMainOutput
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
    ... 26 more

要在本地执行管道,我尝试使用:

  • Maven
  • 数据流SDK

具有以下选项:

数据流 maven

MAVEN dataflow

我应该添加任何依赖项吗?


如果您需要更多详细信息,请告诉我。

1 个答案:

答案 0 :(得分:2)

为解决这个问题,我添加了以下依赖项:

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-direct-java</artifactId>
  <version>${beam.version}</version>
  <scope>runtime</scope>
</dependency>