自动执行DataFlow jar中的Google Cloud Storage的临时位置

时间:2018-03-21 11:17:17

标签: google-cloud-dataflow

我将DataFlow应用程序从Eclipse导出到Runnable JAR文件。 当我运行命令Desktop $ java -jar mariadevconn.jar时,我得到以下异常 为什么命令行的当前工作目录已添加到Google存储路径?谢谢你的帮助。

org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
    at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
    at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
    at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
    at com.masterconcept.cloud.dataflow.connector.MariaDBConnector.main(MariaDBConnector.java:98)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
    ... 4 more
Caused by: java.lang.IllegalArgumentException: Expected a valid 'gs://' path but was given '/Users/henry/Desktop/gs:/hellodataflowfromhenry/temp/staging/'
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:101)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPath(GcsPathValidator.java:75)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:60)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:237)
    ... 9 more
Caused by: java.lang.IllegalArgumentException: Invalid GCS URI: /Users/henry/Desktop/gs:/hellodataflowfromhenry/temp/staging/
    at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191)
    at org.apache.beam.sdk.util.gcsfs.GcsPath.fromUri(GcsPath.java:116)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:99)
    ... 12 more

部分代码:

DataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(argv).withValidation().as(DataflowPipelineOptions.class);
options.setGcpTempLocation("gs://hellodataflowfromhenry/temp");
options.setRunner(DataflowRunner.class);
....

的pom.xml:

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
    <version>2.2.0</version>
    <scope>runtime</scope>
</dependency>
<dependency>
      <groupId>com.google.cloud.dataflow</groupId>
      <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
      <version>2.2.0</version>
</dependency>

工作目录:

Desktop$pwd
/Users/henry/Desktop

0 个答案:

没有答案