Question

我正在学习Apache Spark并试图在Java程序中运行Spark，但是我遇到了一些问题我使用Maven构建程序，我创建了一个新的maven项目。这是pom.xml

  <groupId>spark</groupId>
  <artifactId>testspark</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>testspark</name>
  <url>http://maven.apache.org</url>
  <properties>
   <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  </properties>
  <repositories>
     <repository>
       <id>Akka repository</id>
       <url>http://repo.akka.io/releases</url>
     </repository>
  </repositories>
  <dependencies>
    <dependency>
     <groupId>junit</groupId>
     <artifactId>junit</artifactId>
     <version>3.8.1</version>
     <scope>test</scope>
   </dependency>
   <dependency> 
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.10</artifactId>
     <version>0.9.0-incubating</version>
     </dependency>
  </dependencies>

我在Java的Spark程序就在这里：

public class SimpleApp{ 
    public static void main(String[] args) { 
            String logFile = "/usr/Java/spark-0.9.0/README.md"; 
            JavaSparkContext sc = new JavaSparkContext("local", "Simple App","/usr/Java/spark-0.9.0",new String[] { "target/testspark-0.0.1-SNAPSHOT.jar" }); 
            JavaRDD<String> logData = sc.textFile(logFile).cache(); 
            long numAs = logData.filter(new Function<String, Boolean>() { 
                    public Boolean call(String s) { 
                            return s.contains("a"); 
                    } 
            }).count(); 

            long numBs = logData.filter(new Function<String, Boolean>() { 
                    public Boolean call(String s) { 
                            return s.contains("b"); 
                    } 
            }).count(); 

            System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs); 
    }

}

我使用mvn package and mvn -X exec:java -Dexec.mainClass="spark.testspark.SimpleApp"来运行程序。我可以得到程序结果：Lines with a: 70, lines with b: 44，但在此之后会出现以下问题：

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3:java (default-cli) on project testspark: An exception occured while executing the Java class. null: InterruptedException -> [Help 1] 
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3:java (default-cli) on project testspark: An exception occured while executing the Java class. null 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) 
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108) 
        at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76) 
        at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) 
        at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116) 
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361) 
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) 
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) 
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213) 
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:157) 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
        at java.lang.reflect.Method.invoke(Method.java:606) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) 
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) 
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occured while executing the Java class. null 
        at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:345) 
        at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133) 
        at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) 
        ... 19 more 
Caused by: java.lang.InterruptedException 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) 
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
        at org.apache.spark.scheduler.SparkListenerBus$$anon$2.run(SparkListenerBus.scala:42) 
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles: 
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
root@ubuntu:~/workspace/testspark# mvn -X exec:java -Dexec.mainClass="spark.testspark.SimpleApp"

我对这些问题感到困惑。任何的想法？谢谢！

Answer 1

我认为你忘了在最后调用JavaSparkContext stop（）方法。以下是否也会给你错误？

public class SimpleApp{ 
public static void main(String[] args) { 
        String logFile = "/usr/Java/spark-0.9.0/README.md"; 
        JavaSparkContext sc = new JavaSparkContext("local", "Simple App","/usr/Java/spark-0.9.0",new String[] { "target/testspark-0.0.1-SNAPSHOT.jar" }); 
        JavaRDD<String> logData = sc.textFile(logFile).cache(); 
        long numAs = logData.filter(new Function<String, Boolean>() { 
                public Boolean call(String s) { 
                        return s.contains("a"); 
                } 
        }).count(); 

        long numBs = logData.filter(new Function<String, Boolean>() { 
                public Boolean call(String s) { 
                        return s.contains("b"); 
                } 
        }).count(); 

        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
        sc.stop();
  }
}

一个简单的Spark应用程序，存在MojoExecutionException问题

1 个答案: