无法从Spark作业写入Redis

时间:2018-08-28 11:04:25

标签: json apache-spark redis

我正在创建一个scala哈希图,并尝试通过spark作业在redis中将其编写为Java哈希图,但是却出现此错误:

java.lang.NoClassDefFoundError:
redisson/jackson/databind/jsontype/TypeResolverBuilder
2018-08-28 10:39:48:: at org.redisson.config.Config.<init> . 
(Config.java:101)
2018-08-28 10:39:48:: at org.redisson.Redisson.<init> . 
(Redisson.java:113)
2018-08-28 10:39:48:: at 
org.redisson.Redisson.create(Redisson.java:154)
2018-08-28 10:39:48:: at 
com.oto.sims.writers.RedisWriter$.apply(RedisWriter.scala:237)
2018-08-28 10:39:48:: at com.oto.sims.userfeatureaggregator.usersrecommendation.UserRecommendationAnalyzer$$anonfun$trackRecTypewiseRequests$1.apply(UserRecommendationAnalyzer.scala:158)
2018-08-28 10:39:48:: at com.oto.sims.userfeatureaggregator.usersrecommendation.UserRecommendationAnalyzer$$anonfun$trackRecTypewiseRequests$1.apply(UserRecommendationAnalyzer.scala:155)
2018-08-28 10:39:48:: at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
2018-08-28 10:39:48:: at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
2018-08-28 10:39:48::   at com.oto.sims.engine.UserRecommendationActor$$anonfun$receive$1.applyOrElse(UserRecommendationActor.scala:24)
2018-08-28 10:39:48::   at akka.actor.Actor$class.aroundReceive(Actor.scala:514)
2018-08-28 10:39:48::   at com.oto.sims.engine.UserRecommendationActor.aroundReceive(UserRecommendationActor.scala:16)
2018-08-28 10:39:48::   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
2018-08-28 10:39:48::   at akka.actor.ActorCell.invoke(ActorCell.scala:496)
2018-08-28 10:39:48::   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
2018-08-28 10:39:48::   at akka.dispatch.Mailbox.run(Mailbox.scala:224)
2018-08-28 10:39:48::   at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
2018-08-28 10:39:48::   at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2018-08-28 10:39:48::   at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
2018-08-28 10:39:48::   at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
2018-08-28 10:39:48::   at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-08-28 10:39:48::Caused by: java.lang.ClassNotFoundException: 
redisson.jackson.databind.jsontype.TypeResolverBuilder
2018-08-28 10:39:48::   at 
java.net.URLClassLoader.findClass(URLClassLoader.java:381)
2018-08-28 10:39:48::   at 
java.lang.ClassLoader.loadClass(ClassLoader.java:424)
2018-08-28 10:39:48::   at 
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2018-08-28 10:39:48::   ... 23 more

这是我的Dependencies.scala文件的内容:

val commonDependencies: Seq[ModuleID] = Seq(
"org.scala-lang" % "scala-library" % "2.11.11",
log4j,
"mysql" % "mysql-connector-java" % mysqlVersion,
"com.typesafe" % "config" % "1.2.1",
"org.redisson" % "redisson" % "3.4.2",
"joda-time" % "joda-time" % "2.8",
"com.fasterxml.jackson.core" % "jackson-core" % "2.7.9",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.7.9",
"com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.7.9"

我没有Redis的经验。任何帮助将不胜感激。

1 个答案:

答案 0 :(得分:0)

您提供的详细信息不完整, 您在Redis服务器中使用的端口号 您在哪个OS上运行一台计算机和虚拟机,或不同的计算机。

尽管如此,您可以通过检查以下几点来解决此问题:

-确保您可以ping服务器IP

  • 尝试检查端口是否打开,如果使用Windows OS,则可以在终端Nbtstat上使用telnet。还检查db mysql端口

  • 将jackson lib更新到最新版本或版本号1.9.13,以防新版本不起作用。