Question

我正在使用CombineFn的子类来实现CombinePerKeyExample，而不是使用SerializableFunction的实现

package me.examples;

import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.transforms.Combine.CombineFn;

import java.util.HashSet;
import java.util.Set;

public class ConcatWordsCombineFn extends CombineFn<String, ConcatWordsCombineFn.Accumulator, String> {
    @DefaultCoder(AvroCoder.class)
    public static class Accumulator{
        HashSet<String> plays;
    }

    @Override
    public Accumulator createAccumulator(){
        Accumulator accumulator = new Accumulator();
        accumulator.plays = new HashSet<>();
        return accumulator;
    }

    @Override
    public Accumulator addInput(Accumulator accumulator, String input){
        accumulator.plays.add(input);
        return accumulator;
    }

    @Override
    public Accumulator mergeAccumulators(Iterable<Accumulator> accumulators){
        Accumulator mergeAccumulator = new Accumulator();
        mergeAccumulator.plays = new HashSet<>();

        for(Accumulator accumulator: accumulators){
            mergeAccumulator.plays.addAll(accumulator.plays);
        }

        return mergeAccumulator;
    }

    @Override
    public String extractOutput(Accumulator accumulator){
        //how to access the key here ? 
        return String.join(",", accumulator.plays);
    }
}

管道由ReadFromBigQuery，ExtractAllPlaysOfWords（下面的代码）和WriteToBigQuery

组成

package me.examples;

import com.google.api.services.bigquery.model.TableRow;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;

public class PlaysForWord extends PTransform<PCollection<TableRow>, PCollection<TableRow>> {


    @Override
    public PCollection<TableRow> expand(PCollection<TableRow> input) {

            PCollection<KV<String, String>> largeWords = input.apply("ExtractLargeWords", ParDo.of(new ExtractLargeWordsFn()));
            PCollection<KV<String, String>> wordNPlays = largeWords.apply("CombinePlays",Combine.perKey(new ConcatWordsCombineFn()));
            wordNPlays.setCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()));
            PCollection<TableRow> rows = wordNPlays.apply("FormatToRow", ParDo.of(new FormatShakespeareOutputFn()));
            return rows;
    }
}

我想访问ConcatWordsCombineFn中的密钥，以便以此为基础进行最终累加。例如，如果键以,开头，则用a连接单词，否则使用;。

查看编程指南时

如果您需要根据密钥更改组合策略（例如，某些用户的MIN，其他用户的MAX），则可以定义一个KeyedCombineFn 来访问组合策略中的密钥

我在KeyedCombineFn中找不到org.apache.beam.sdk.transforms.Combine 我正在使用Apache Beam 2.12.0和Google Dataflow作为运行程序。

Answer 1

我认为没有内置的方法可以解决此问题。最直接的解决方法（我知道这并不完美）是将字符串包装到另一个KV：KV<String, KV<String, String>>中，两个键都相同。

合并KV对的PCollection时，如何访问CombinerFn子类中的密钥？

1 个答案: