Question

我尝试使用Apache Beam 0.6.0在GCP上启动Dataflow作业。我正在使用shade插件编译一个超级jar，因为我无法使用＆＃34; mvn：execjava＆＃34;来启动作业。我包括这种依赖：

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
  <version>0.6.0-SNAPSHOT</version>
</dependency>

我收到以下异常：

Exception in thread "main" java.lang.IllegalArgumentException: Unknown 'runner' specified 'DataflowRunner', supported pipeline runners [DirectRunner]
    at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1609)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:104)
    at org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:289)
    at com.disney.dtss.desa.tools.SpannerSinkTest.main(SpannerSinkTest.java:116)
Caused by: java.lang.ClassNotFoundException: DataflowRunner
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:264)
    at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1595)

我错过了其他什么吗？

Answer 1

尝试

mvn compile exec:java -Dexec.mainClass=Yourmain Class -Pdataflow-runner

*在最后添加-Pdataflow-runner

Answer 2

在@Andrew Nguonly's comment之后，我将en = df["Blk_end"].values.astype('datetime64[D]') st = df["Blk_start"].values.astype('datetime64[D]') df["x"] = np.busday_count(st, en)的依赖项复制到DataflowRunner文件中的外部作用域（复制到<dependencies>标记中）。

基本上添加了这个：

pom.xml

在波束wordCount示例的<dependency> <groupId>org.apache.beam</groupId> <artifactId>beam-runners-google-cloud-dataflow-java</artifactId> <version>${beam.version}</version> <scope>runtime</scope> </dependency>处</dependencies>结束之前。

尝试使用DataflowRunner时出现ClassNotFound异常

2 个答案: