Question

嗨，我想在Dataflow Pipelnie中动态创建表

首先，我捕获BigQueryIO WriteResult，然后使用它创建表

    <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.springboot.application</groupId>
    <artifactId>springbootmvc2</artifactId>
    <packaging>war</packaging>
    <version>0.0.1-SNAPSHOT</version>
    <name>springbootmvc2 Maven Webapp</name>
    <url>http://maven.apache.org</url>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.9.RELEASE</version>
    </parent>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-tomcat</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.tomcat.embed</groupId>
            <artifactId>tomcat-embed-jasper</artifactId>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>jstl</artifactId>
        </dependency>
    </dependencies>        
</project>

但是我仍然遇到未找到表的异常

        WriteResult writeResult =
            incomingRecords.apply(
                    "WriteToBigQuery",
                    BigQueryIO.<TableRowWithSchema>write()
                            .to(new DynamicTables())
                            .withFormatFunction(TableRowWithSchema::getTableRow)
                            .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
                            .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
                            .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors()));

有什么问题吗？谢谢

Answer 1

您错了两件事：

1）如果要动态创建表，则需要使用CREATED_IF_NEEDED处理。

withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)

2）动态创建表时，您需要使用withSchema方法或withJsonSchema方法传递模式。

那可以解决问题！

使用数据流失败插入WriteResult来处理未找到表的异常

1 个答案: