Question

Am using Google Data Flow where in one of the steps am subscribing to a topic in pub sub using already created subscription. Here is the code snippet

CustomPipelineOptions options =
            PipelineOptionsFactory.fromArgs(args).withValidation().as(customPipelineOptions.class);
    Pipeline p = Pipeline.create(options);

    PCollection<TableRow> datastream = p.apply(PubsubIO.Read.named("Read device  data from PubSub")                 .subscription("projects/<projectID>/subscriptions/<subscriptionname>)
            .topic(String.format("projects/%s/topics/%s", options.getSourceProject(), options.getSourceTopic()))
            .timestampLabel("ts")
            .withCoder(TableRowJsonCoder.of()));

The above code when executed results in the following error: Error processing pipeline. Causes: (b5e276ef8c76419f): Unrecognized input pubsub_subscription for step s1.

Am passing the right subscription name and project ID. Not sure why am still getting the above error.

Please kindly help.

Answer 1

指定2个来源中的一个应该足够：主题或订阅。

我建议你试试：

PCollection<TableRow> datastream = p
        .apply(PubsubIO.Read.named("Read device data from PubSub")
        .topic(String.format("projects/%s/topics/%s", options.getSourceProject(), options.getSourceTopic()))
        .timestampLabel("ts")
        .withCoder(TableRowJsonCoder.of()));

另外：我想您使用的是Dataflow 1.9 SDK？您可能想考虑转移到new Beam 2.0.0 release。您可以在该SDK here中找到PubSub的参考。

Using existing pub sub subscription from google data flow

1 个答案: