代码

#!/usr/bin/env

import apache_beam as beam
from apache_beam.io.textio import WriteToText

def addoutput(line):
    return [line, "Its weekend!"]

with beam.Pipeline() as p:
    ( p
      | beam.Create(["blah"])
      | beam.Map(addoutput)
      | WriteToText(file_path_prefix="/tmp/sample")
    )

结果

['blah', 'Its weekend!']

但是，当我尝试使用java做同样的事情时，我在 maven 中遇到了编译错误。

代码

public class SampleTextIO
{
    static class AddFieldFn extends DoFn<String, String> {

        @ProcessElement
        public void processElement(@Element String word, OutputReceiver<String> receiver) {

            receiver.output(word);
            receiver.output("Its weekend!");
        }
    }

    public static void main ( String[] args ) {
        System.out.println( "Main class for DirectRunner" );

        // Pipeline create using default runner (DirectRunnter)
        // Interface: PipelineOptions
        PipelineOptions options = PipelineOptionsFactory.create();

        Pipeline p = Pipeline.create(options);

        // Example pcollection
        final List<String> LINES = Arrays.asList(
            "blah"
        );

        // Read lines from file
        p.apply(Create.of(LINES))
         .apply(MapElements.via(new AddFieldFn()))
         .apply(TextIO.write().to("/tmp/test-out"));

        p.run().waitUntilFinish();
    }
}

结果

[ERROR] /home/ywatanabe/git/google-data-engineer/Data_Science_on_the_Google_Cloud_Platform/Ch04/java/directrunner/src/main/java/com/example/SampleTextIO.java:[43,28] no suitable method found for via(com.example.SampleTextIO.AddFieldFn)
[ERROR]     method org.apache.beam.sdk.transforms.MapElements.<InputT,OutputT>via(org.apache.beam.sdk.transforms.InferableFunction<InputT,OutputT>) is not applicable
[ERROR]       (cannot infer type-variable(s) InputT,OutputT
[ERROR]         (argument mismatch; com.example.SampleTextIO.AddFieldFn cannot be converted to org.apache.beam.sdk.transforms.InferableFunction<InputT,OutputT>))
[ERROR]     method org.apache.beam.sdk.transforms.MapElements.<InputT,OutputT>via(org.apache.beam.sdk.transforms.SimpleFunction<InputT,OutputT>) is not applicable
[ERROR]       (cannot infer type-variable(s) InputT,OutputT
[ERROR]         (argument mismatch; com.example.SampleTextIO.AddFieldFn cannot be converted to org.apache.beam.sdk.transforms.SimpleFunction<InputT,OutputT>))
[ERROR]     method org.apache.beam.sdk.transforms.MapElements.via(org.apache.beam.sdk.transforms.ProcessFunction) is not applicable
[ERROR]       (argument mismatch; com.example.SampleTextIO.AddFieldFn cannot be converted to org.apache.beam.sdk.transforms.ProcessFunction)
[ERROR]     method org.apache.beam.sdk.transforms.MapElements.via(org.apache.beam.sdk.transforms.SerializableFunction) is not applicable
[ERROR]       (argument mismatch; com.example.SampleTextIO.AddFieldFn cannot be converted to org.apache.beam.sdk.transforms.SerializableFunction)
[ERROR]     method org.apache.beam.sdk.transforms.MapElements.via(org.apache.beam.sdk.transforms.Contextful) is not applicable
[ERROR]       (argument mismatch; com.example.SampleTextIO.AddFieldFn cannot be converted to org.apache.beam.sdk.transforms.Contextful)

阅读javadoc， MapElements 支持 Processfunction ，但在我的情况下效果不佳。

如何动态在 java 中添加 python 之类的字段？

Answer 1

这是因为via的{{1}}方法期望以下之一：mapElements，InferableFunction，SimpleFunction，ProcessFunction，{ {1}}。在您的示例中，SerializableFunction扩展了Contextful。另外，与Python示例相比，您似乎希望输出两个元素的列表，而不是产生两个不同的行。

有关如何执行此操作的三个示例：

AddFieldFn

其中DoFn是：

// via ProcessFunction
PCollection p1 = p.apply(Create.of(LINES))
  .apply(MapElements.into(TypeDescriptors.lists(TypeDescriptors.strings()))
                    .via((String word) -> (Arrays.asList(word, "Its weekend!"))))
  .apply(ParDo.of(new PrintResultsFn()));

// via in-line SimpleFunction
PCollection p2 = p.apply(Create.of(LINES))
  .apply(MapElements.via(new SimpleFunction<String, List<String>>() {
    public List<String> apply(String word) {
      return Arrays.asList(word, "Its weekend!");
    }}))
  .apply(ParDo.of(new PrintResultsFn()));

// via AddFieldFn class 
PCollection p3 = p.apply(Create.of(LINES))
  .apply(MapElements.via(new AddFieldFn()))
  .apply(ParDo.of(new PrintResultsFn()));

和AddFieldFn验证行：

// define AddFieldFn extending from SimpleFunction and overriding apply method
static class AddFieldFn extends SimpleFunction<String, List<String>> {
    @Override
    public List<String> apply(String word) {
        return Arrays.asList(word, "Its weekend!");
    }
}

应打印所需的输出：

PrintResultsFn

完整代码here。经过DirectRunner和Java SDK 2.13.0的测试

如何在apache_beam中的MapElements中动态添加字段？

代码

结果

代码

结果

1 个答案: