Playframework和Spark类路径问题

时间:2017-04-14 12:38:52

标签: classpath apache-spark-sql playframework-2.5

此问题是否有解决方法?有人设法在开发模式下将spark集成到playframework中吗?

我正在运行playframework-2.5应用程序,该应用程序使用spark-sql-2.1.0从csv文件中读取。应由spark-sql实例化的类是Listing2Exported

我读到在dev模式下运行时可能会出现一些类路径问题,我可以确认一下。

  • dev模式:如果我执行activator run,我会得到例外
  • prod mode:如果我执行activator start,那么文件读取正确。

case class Listing2Exported (
    extractor:Option[String],
    categoryId:Option[Int],
    categoryName:Option[String],
    productName:Option[String],
    productCode:Option[Int]
)

例外

1) Error injecting constructor, scala.ScalaReflectionException: class org.raisercostin.namek.io.Listing2Exported in JavaMirror with PlayDependencyClassLoader{.... ] not found.
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123) ~[scala-reflect-2.11.8.jar:na]
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22) ~[scala-reflect-2.11.8.jar:na]
    at org.raisercostin.namek.io.ListingSparkCsvImporter$$typecreator1$1.apply(ListingSparkCsvImporter.scala:70) ~[na:na]
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232) ~[scala-reflect-2.11.8.jar:na]
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232) ~[scala-reflect-2.11.8.jar:na]
    at org.apache.spark.sql.catalyst.ScalaReflection$class.localTypeOf(ScalaReflection.scala:782) ~[spark-catalyst_2.11-2.1.0.jar:2.1.0]
    at org.apache.spark.sql.catalyst.ScalaReflection$.localTypeOf(ScalaReflection.scala:39) ~[spark-catalyst_2.11-2.1.0.jar:2.1.0]
    at org.apache.spark.sql.catalyst.ScalaReflection$.dataTypeFor(ScalaReflection.scala:63) ~[spark-catalyst_2.11-2.1.0.jar:2.1.0]
    at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$.apply(ExpressionEncoder.scala:63) ~[spark-catalyst_2.11-2.1.0.jar:2.1.0]
    at org.apache.spark.sql.Encoders$.product(Encoders.scala:275) ~[spark-catalyst_2.11-2.1.0.jar:2.1.0]
    at org.raisercostin.namek.io.ListingSparkCsvImporter.readCsv(ListingSparkCsvImporter.scala:70) ~[na:na]
    at org.raisercostin.namek.io.ListingSparkCsvImporter.read(ListingSparkCsvImporter.scala:40) ~[na:na]
    at namek.app.Application$AppDataReader.read2(Application.scala:154) ~[na:na]
    at namek.app.Application$AppDataReader$$anonfun$11.apply(Application.scala:157) ~[na:na]
    at namek.app.Application$AppDataReader$$anonfun$11.apply(Application.scala:156) ~[na:na]
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[scala-library-2.11.8.jar:na]
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[scala-library-2.11.8.jar:na]
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) ~[scala-library-2.11.8.jar:na]
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) ~[scala-library-2.11.8.jar:na]
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) ~[scala-library-2.11.8.jar:na]
    at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[scala-library-2.11.8.jar:na]
    at namek.app.Application$AppDataReader.<init>(Application.scala:156) ~[na:na]
    at namek.app.Application$Application.appData$lzycompute(Application.scala:72) ~[na:na]
    at namek.app.Application$Application.appData(Application.scala:71) ~[na:na]
    at namek.app.Application$Application.filter(Application.scala:79) ~[na:na]
    at namek.app.Application$Application.namek$app$Application$Application$$topListings$1(Application.scala:46) ~[na:na]
    at namek.app.Application$Application$$anonfun$2.apply(Application.scala:49) ~[na:na]
    at namek.app.Application$Application$$anonfun$2.apply(Application.scala:49) ~[na:na]
    at models.TopCategory.<init>(Models.scala:160) ~[na:na]
    at namek.app.Application$Application.<init>(Application.scala:49) ~[na:na]
    at namek.services.DefaultListingsService.reloadApplication(ListingsService.scala:63) ~[na:na]
    at namek.services.DefaultListingsService.<init>(ListingsService.scala:43) ~[na:na]
    at namek.services.DefaultListingsService$$FastClassByGuice$$97a4e1c7.newInstance(<generated>) ~[na:na]
    at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) ~[guice-4.0.jar:na]
    at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) ~[guice-4.0.jar:na]
    at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[guice-4.0.jar:na]
    at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) ~[guice-4.0.jar:na]
    at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) ~[guice-4.0.jar:na]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103) ~[guice-4.0.jar:na]
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) ~[guice-4.0.jar:na]
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145) ~[guice-4.0.jar:na]
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) ~[guice-4.0.jar:na]
    at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016) ~[guice-4.0.jar:na]
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) ~[guice-4.0.jar:na]
    at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012) ~[guice-4.0.jar:na]
    ... 47 common frames omitted

0 个答案:

没有答案