Question

使用apex / spark runner将数据写入Apache beam中的表格。但是在使用apex runner运行程序时遇到异常。

 List<TableFieldSchema> fields = new ArrayList<>();
            fields.add(new TableFieldSchema().setName("Id").setType("STRING"));
            fields.add(new TableFieldSchema().setName("row").setType("STRING"));
            TableSchema schema = new TableSchema().setFields(fields);

PCollection<TableRow> data= pipeline.apply("ReadLines", TextIO.read().from(inputDir + "data.txt"))
        .apply(ParDo.of(new ExtractDataFn()));

data.apply(BigQueryIO.writeTableRows()
                 .to("my-project:output.output_table")
                 .withSchema(schema)
                 .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
                 .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_EEDED));

static class ExtractDataFn extends DoFn<String, TableRow> {
    /**
     * 
     */
    private static final long serialVersionUID = 1L;

    @ProcessElement
    public void processElement(ProcessContext c) {

        if (c.element() != null) {
            TableRow row = new TableRow().set("Id", c.element().substring(1, 5)).set("row", c.element());
            c.output(row);
        }
    }
}

使用命令 mvn执行上述程序后执行exec：java ... --runner = ApexRunner＆＃34; -Papex-runner ，低于例外：

 java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
        at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expand(BigQueryIO.java:1398)
        at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write.expand(BigQueryIO.java:974)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:525)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:460)
        at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:284)

你能指导我一下会出现什么问题

Answer 1

这是一个常见错误，因为您的程序（包括Beam SDK）和Apex运行时环境都包括Guava，显然是冲突的版本。典型的答案是，如果可能，您应该更改您的Guava依赖项版本，或者在Maven中对其进行着色。

有关如何处理其中每一项的更详细说明，请搜索＆＃34; java.lang.NoSuchMethodError：com.google.common.base.Preconditions＆＃34;。

Apache Beam -BigQueryIO使用Apex runner

1 个答案: