Question

我试图在远程群集上运行我的Spark应用程序，并且我遇到了序列化错误。 Scala和Spark版本是一样的。我已经陷入困境。

群集上的spark-shell -version：

root@a913008dd071:/usr/local/spark-2.1.1# ./bin/spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
Branch
Compiled by user jenkins on 2017-04-25T23:51:10Z
Revision
Url
Type --help for more information.

build.sbt

import sbt.ExclusionRule

name := "hxfa"
version := "1.0"
scalaVersion := "2.11.8"

val elasticVersion = "5.4.1"

resolvers += "Spark Packages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += "Additional spark packages" at "https://dl.bintray.com/sbcd90/org.apache.spark"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/"
resolvers += "Spring Plugins" at "http://repo.spring.io/plugins-release/"

/* Dependencies */
libraryDependencies ++= Seq(
  // Framework and configuration
  "org.springframework.boot" % "spring-boot-starter-web" % "1.5.4.RELEASE",
  "org.hibernate" % "hibernate-validator" % "5.2.4.Final",

  /* Serializations */
  "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7",
  "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7",
  "com.esotericsoftware" % "kryo" % "4.0.0",


  // Spark and utilities
  "org.apache.spark" %% "spark-core" % "2.1.0",
  "org.apache.spark" %% "spark-sql" % "2.1.0" ,
  "org.apache.spark" %% "spark-mllib" % "2.1.0" ,
  "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11",


  // Spark connectors
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % elasticVersion,
  "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0",


  //JDBC
  "mysql" % "mysql-connector-java" % "5.1.35",

  // HBase
  "org.apache.hbase" % "hbase" % "1.2.4",
  "org.apache.hbase" % "hbase-client" % "1.2.4",
  "org.apache.hbase" % "hbase-common" % "1.2.4",

  // OrientDB
  "com.orientechnologies" % "orientdb-graphdb" % "2.2.20"

).map(_.excludeAll(ExclusionRule("org.slf4j", "slf4j-log4j12"), ExclusionRule("log4j", "log4j")))

libraryDependencies ++= Seq(
  "org.apache.hbase" % "hbase-server" % "1.2.4"
).map(_.excludeAll(
  ExclusionRule("com.sun.jersey", "jersey-server"),
  ExclusionRule("tomcat"),
  ExclusionRule("log4j", "log4j")
))


/* Assembly     */

mainClass in assembly := Some("com.x.x.hello.app.HX")
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false)

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs@_*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

堆栈追踪：

java.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;本地类不兼容：stream classdesc serialVersionUID = -2992553500466442037，本地类serialVersionUID = -5670082246090726217 在java.io.ObjectStreamClass.initNonProxy（ObjectStreamClass.java:616）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readNonProxyDesc（ObjectInputStream.java:1843）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readClassDesc（ObjectInputStream.java:1713）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2000）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] 在scala.collection.immutable.List $ SerializationProxy.readObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] 在java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] 在java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] 在scala.collection.immutable.List $ SerializationProxy.readObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] 在java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] 在java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] 在scala.collection.immutable.List $ SerializationProxy.readObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] 在java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] 在java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] 在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] 在java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] 在org.apache.spark.serializer.JavaDeserializationStream.readObject（JavaSerializer.scala：75）〜[spark-core_2.11-2.1.0.jar：2.1.0] 在org.apache.spark.serializer.JavaSerializerInstance.deserialize（JavaSerializer.scala：114）〜[spark-core_2.11-2.1.0.jar：2.1.0] 在org.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：80）〜[spark-core_2.11-2.1.0.jar：2.1.0] 在org.apache.spark.scheduler.Task.run（Task.scala：99）〜[spark-core_2.11-2.1.0.jar：2.1.0] 在org.apache.spark.executor.Executor $ TaskRunner.run（Executor.scala：322）〜[spark-core_2.11-2.1.0.jar：2.1.0] 在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）〜[na：1.8.0_45] at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617)~ [na：1.8.0_45] 在java.lang.Thread.run（Thread.java:748）〜[na：1.8.0_45]

Answer 1

spark-submit --version显示其软件包的spark和scala版本，而不是系统，而你的sbt正在使用系统的scala版本。所以

请更改

  "org.apache.spark" %% "spark-core" % "2.1.0",
  "org.apache.spark" %% "spark-sql" % "2.1.0" ,
  "org.apache.spark" %% "spark-mllib" % "2.1.0" ,

到

  "org.apache.spark" % "spark-core_2.11" % "2.1.1",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" ,
  "org.apache.spark" % "spark-mllib_2.11" % "2.1.1" ,

如果它没有帮助，请使用您的系统scala版本更新您的问题，如何提交应用程序以及您的远程计算机的scala和spark版本。

Spark java.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;本地类不兼容

1 个答案: