Question

我在执行以下代码时遇到错误：

tableRows2.apply(BigQueryIO.writeTableRows()
          .to(new DynamicDestinations<TableRow, TableRow>() {
            private static final long serialVersionUID = 1L;
            @Override
                public TableDestination getTable(TableRow dest) {
                      List<TableRow> list = sideInput(bqDataView);
                      String table = list.get(0).get("table").toString();
                      String tableSpec = "ProjectId:DatasetId."+table;
                      String tableDescription = "";
                      return new TableDestination(tableSpec, tableDescription);
                }

                public String getSideInputs(PCollectionView<List<TableRow>> bqDataView) {
                    String str = bqDataView.toString();
                    return str;
                }

                @Override
                public TableSchema getSchema(TableRow destination) {
                    List<TableRow> list = sideInput(bqDataView);
                    String[] schemas = list.get(0).get("schema").toString().split(",");
                    List<TableFieldSchema> fields = new ArrayList<>();
                    for(int i=0;i<schemas.length;i++)
                    {
                        fields.add(new TableFieldSchema().setName(schemas[i].split(":")[0]).setType(schemas[i].split(":")[1]));
                    }
                    TableSchema schema = new TableSchema().setFields(fields);
                    return schema;
                }

                @Override
                public TableRow getDestination(ValueInSingleWindow<TableRow> element) {
                    return null;
                }
              }.getSideInputs(bqDataView)).withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER));

我得到的错误是：

  (7dc1af5b557d4d6b): java.lang.IllegalArgumentException: Table reference is not in [project_id]:[dataset_id].[table_id] format: SimplePCollectionView{tag=Tag<org.apache.beam.sdk.values.PCollectionViews$SimplePCollectionView.<init>:403#5f2ef1f005ae0b4>}
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers.parseTableSpec(BigQueryHelpers.java:102)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$TableSpecToTableRef.apply(BigQueryHelpers.java:286)
    at org.apache.beam.sdk.io.gcp.bigquery.BigQueryHelpers$TableSpecToTableRef.apply(BigQueryHelpers.java:282)
    at org.apache.beam.sdk.options.ValueProvider$NestedValueProvider.get(ValueProvider.java:131)
    at org.apache.beam.sdk.options.ValueProvider$NestedValueProvider.get(ValueProvider.java:131)
    at org.apache.beam.sdk.options.ValueProvider$NestedValueProvider.get(ValueProvider.java:131)
    at org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinationsHelpers$ConstantTableDestinations.getDestination(DynamicDestinationsHelpers.java:64)
    at org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinationsHelpers$ConstantTableDestinations.getDestination(DynamicDestinationsHelpers.java:41)
    at org.apache.beam.sdk.io.gcp.bigquery.PrepareWrite$1.processElement(PrepareWrite.java:58)

这是在Apache Beam中实现DynamicDestinations的正确方法吗？另外，是否总是需要在getSideInputs（）方法中返回String？应该在getDestination（）方法中写什么？

感谢。

Answer 1

我认为您错误地将旁边输入传递给DynamicDestinations。具体来说，由于您在getSideInputs的调用中呼叫to(...)，因此您使用getSideInputs的结果 - 基本上如下：

tableRows2.apply(BigQueryIO.writeTableRows()
      .to(bqDataView.toString())

查看DynamicDestinations的界面和BigQueryIO的测试，看起来你应该做更多类似的事情：

final PCollectionView<List<TableRow>> bqDataView = /* ... */;
tableRows2.apply(BigQueryIO.writeTableRows()
    .to(new DynamicDestinations<TableRow, TableRow>() {

        // ...

        // Note this method needs to be overridden and use
        // the same signature as in DynamicDestinations. Also,
        // it should not be invoked as part of the apply.
        @Override
        public List<PCollectionView<?>> getSideInputs() {
          return ImmutableList.of(bqDataView);
        }

        // ...
      }).withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER));

此外，第二种类型应该是您可以分组以识别要写入的表，而不是使用DynamicDestinations<TableRow, TableRow>。在您的情况下，似乎所有内容都将转到相同的实际目的地，因此您可以使用具有固定值的String：

new DynamicDestinations<TableRow, String>() {
  @Override
  public TableDestination getTable(String unusedDest) {
    // ...
  }

  @Override
  public TableSchema getSchema(String unusedDest) {
    ...
  }

  @Override
  public String getDestination(ValueInSingleWindow<TableRow> element) {
    return "destination";
  }
}

DynamicDestinations中的错误：Apache Beam

1 个答案: