Question

使用Spark作为Runner，我在Apache Beam中的Java中遇到以下异常。我评论了错误的话。

PipelineOptions o = PipelineOptionsFactory.create();
SparkContextOptions options = o.as(SparkContextOptions.class);

SparkConf conf = new SparkConf().setMaster("local[4]").setAppName("test");
JavaSparkContext jsc = new JavaSparkContext(conf);

options.setRunner(SparkRunner.class);
options.setUsesProvidedSparkContext(true);
options.setProvidedSparkContext(jsc);

Pipeline pipeline = Pipeline.create(options); //Error in this line

上面是代码，下面是错误！！

3639 [main] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@18324f97{/metrics/json,null,AVAILABLE,@Spark} Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method SparkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233) at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162) at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150) at com.svayamtech.bean_app.Wordcount_lambda.main(Wordcount_lambda.java:42) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222) ... 4 more Caused by: java.lang.NoClassDefFoundError: org/apache/beam/runners/core/construction/PipelineResources at org.apache.beam.runners.spark.SparkRunner.fromOptions(SparkRunner.java:129) ... 9 more Caused by: java.lang.ClassNotFoundException: org.apache.beam.runners.core.construction.PipelineResources at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 10 more 3690 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook 3696 [Thread-1] INFO org.spark_project.jetty.server.AbstractConnector - Stopped Spark@78f9ed3e{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

Spark Runner无法正常工作？有人可以帮我解决这个问题吗？

Answer 1

尝试在您的SparkContext中添加所有不透明的罐子。

SparkConf conf = new SparkConf().setMaster("local[4]")
  .setAppName("test").setJars(Seq("beam-runners-core-construction-java-2.5.0.jar"));

无法从工厂方法SparkRunner＃fromOptions构造实例

1 个答案: