Question

对于BigQuery批处理管道，模板只能执行一次，因为BigQuery作业ID是在模板创建时设置的。我正在使用Apache beam v2.0.0并且无法多次执行模板。我们可以在头部使用光束进行此限制吗？如果是的话，我想知道的第一件事就是梁头？我的Apache Beam程序需要多次支持模板执行的确切更改？

Maven依赖：

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-jms</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-examples-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-examples-java8</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-common-fn-api</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-build-tools</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-core</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-join-library</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-protobuf</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-extensions-sorter</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-amqp</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-jdbc</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-kafka</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-kinesis</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-mongodb</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-mqtt</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-solr</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-core-construction-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-core-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-direct-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-common-runner-api</artifactId>
  <version>2.2.0-SNAPSHOT</version>
</dependency>

Answer 1

这是问题BEAM-2058。如果您使用Beam github repository中的最新代码，则应该修复此问题。除了构建新版本的Beam并更新你的pom.xml以使用它之外，你不应该做任何事情。

或者，等待正在准备的Beam的2.1.0版本。

使用BigQuery作为接收器多次执行模板

1 个答案: