Question

我尝试在我的单元测试中使用非常简单的输入参数运行我的应用程序，如下所示，但它不起作用。

class MyApplicationTest {

  @Test
  def testMyApplication = {

    val args: Array[String] = Array()
    MyApplication.main(args)
  }
}

MyApplication.scala是Spark工作。我可以在eclipse中将其作为Scala应用程序运行。我只是想从单元测试中调用它。以下是我收到的错误消息，有人可以帮帮我吗？谢谢！

错误似乎来自这一行：

val sc = new SparkContext(conf)

以下是追踪：

java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
    at java.lang.ClassLoader.checkCerts(ClassLoader.java:952)
    at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:794)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:136)
    at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:129)
    at org.spark-project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:98)
    at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:101)
    at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:92)
    at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:78)
    at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62)
    at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:62)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:62)
    at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:50)
    at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:61)
    at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:151)
    at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:106)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:300)
    at com.inflection.rl.ml.profile.jobs.RecordLinkageScalaJob$.main(RecordLinkageScalaJob.scala:101)
    at com.inflection.rl.ml.profile.jobs.RecordLinkageScalaJobTest.testMyApplication(RecordLinkageScalaJobTest.scala:24)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Answer 1

实际上只是依赖问题。我在pom.xml中添加了以下内容，代码可以正常工作。

    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.4.0</version>
        <exclusions>
            <exclusion>
                <groupId>javax.servlet</groupId>
                <artifactId>servlet-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

使用简单参数从单元测试中调用main（spark）应用程序

1 个答案: