如何使用Spark-Scala在Redis之间保存和检索HashMap？

Question

我正在使用Scala 2.11.12，Spark 2.3.0，Redis 3.2.12。

我正在使用spark-submit命令执行Spark应用程序：

spark-submit \
  --master local \
  --driver-memory 512m \
  --executor-memory  512m \
  --jars /home/ec2-user/jars/redisclient_2.9.1-2.9.jar \
  --class com.ntt.test.TestRedis \
  /home/ec2-user/sparkTesting/testing_2.11-0.1.jar

这给了我以下例外：

Exception in thread "main" 
    java.lang.NoClassDefFoundError:scala/ScalaObject
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)

引用自https://github.com/debasishg/scala-redis

当我尝试不使用罐子执行

/home/ec2-user/jars/redisclient_2.9.1-2.9.jar.

spark-submit --master local --driver-memory 512m --executor-memory 512m --class com.ntt.test.TestRedis /home/ec2-user/sparkTesting/testing_2.11-0.1.jar

    SPARK_MAJOR_VERSION is set to 2, using Spark2
    Exception in thread "main" java.lang.NoClassDefFoundError:com/redis/RedisClient
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:906)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: com.redis.RedisClient
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 11 more

代码

import org.apache.log4j.{Level, Logger}
import org.apache.spark.sql.SparkSession

import com.redis._
object TestRedis extends App { Logger.getLogger("org").setLevel(Level.ERROR)
Logger.getLogger("akka").setLevel(Level.ERROR)

val spark = SparkSession.builder
.appName("Redis-Test")
.config("spark.serializer","org.apache.spark.serializer.KryoSerializer")
.getOrCreate()


val redisServerDnsAddress = "127.0.0.1"
val redisPortNumber = 6379
val r = new RedisClient(redisServerDnsAddress,redisPortNumber)
r.hmset("oeehash", Map("Availability" -> 3.4, "Performance"-> 2.6))
val hs=r.hgetall1("oeehash")
println(hs)
val ava=r.hget("oeehash","Availability")
println("Avail :"+ava)

build.sbt

name := "Testing"
version := "0.1"
scalaVersion := "2.11.12"
val sparkVersion = "2.3.0"
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % sparkVersion,"org.apache.spark" %% "spark-sql" %sparkVersion,"org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion)
libraryDependencies ++= Seq("net.debasishg" %% "redisclient" % "3.10")

spark-submit --master local --driver-memory 512m --executor-memory 512m --jars /home/ec2-user/jars/redisclient_2.9.1-2.9.jar --class com.ntt.test.TestRedis /home/ec2-user/sparkTesting/testing_2.11-0.1.jar

它应该给 一些（地图（可用性-> 3.4，性能-> 2.6）） 可用：有些（3.4）

但是下面给出了例外：

        Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)