按照link中的指示,尝试构建JAR文件来安排Airflow中的数据流作业。在依赖关系部分中添加了相关的依赖关系,并在POM文件的构建部分中添加了主类名称。当我使用下面的maven命令执行数据流作业时,出现下面的错误。
有人可以指导我解决这个问题吗?
Maven命令
- mvn package
- java -jar target/sample-1.0.0.jar
错误消息:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/options/PipelineOptions
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.privateGetMethodRecursive(Unknown Source)
at java.lang.Class.getMethod0(Unknown Source)
at java.lang.Class.getMethod(Unknown Source)
at sun.launcher.LauncherHelper.validateMainClass(Unknown Source)
at sun.launcher.LauncherHelper.checkAndLoadMain(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.options.PipelineOptions
我的管道文件
public interface DefaultOptions extends PipelineOptions,GcpOptions,ApplicationNameOptions,DataflowPipelineDebugOptions,DataflowPipelineWorkerPoolOptions,
BigQueryOptions,GcsOptions,StreamingOptions,CloudDebuggerOptions,DataflowProfilingOptions,PubsubOptions {
我在类文件中的管道初始化
PipelineOptionsFactory.register(DefaultOptions.class);
DefaultOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(DefaultOptions.class);
options.setRunner(DataflowRunner.class);
options.setProject(options.getDataFlowProjectName());
options.setTempLocation(options.getDataFlowProjectTempLocation());
options.setGcpTempLocation(options.getDataFlowProjectTempLocation());
Pipeline p = Pipeline.create(options);
POM文件:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.11.0</version>
</dependency>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>SampleWindowJar</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
答案 0 :(得分:0)
Cloud Dataflow依赖性不是您唯一需要的依赖性。您还需要所有与Beam SDK相关的依赖项。获得它们的最简单方法是使用Beam提供的Maven原型:
https://beam.apache.org/get-started/quickstart-java/
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter \
-DarchetypeVersion=2.11.0 \
-DgroupId=org.yourorg \
-DartifactId=my-beam-pipeline \
-Dversion="0.1" \
-Dpackage=org.yourorg.pipelines \
-DinteractiveMode=false
cd my-beam-pipeline
您会看到一个pom.xml
文件,以及一个可以开发的包含启动程序管道的目录。
一旦创建了原型,它将包括所有基本的SDK依赖关系,还可以添加Dataflow Runner:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.11.0</version>
</dependency>