如何解决Scala -Spark中的java.io.invalid类异常SerialVersion UID

时间：2015-12-29 07:17:57

标签： scala apache-spark hive

我一直试图解决这个例外很多天但无法解决这个问题。 细节： 使用Scala尝试从其他群集中的Hive表连接和检索数据 在本地计算机和Spark节点中使用Scala版本-2.10.4 在两者上使用Spark 1.4.1（使用maven依赖项在项目中添加）和

上的JVM 1.7

尝试使用以下代码

    @SerialVersionUID(42L)
object CountWords extends serializable {

  def main(args:Array[String]){
    println("hello")    
    val objConf = new SparkConf().setAppName("Spark Connection").setMaster("spark://IP:7077")    
    var sc = new SparkContext(objConf)    
    val objHiveContext = new HiveContext(sc)
    objHiveContext.sql("USE MyDatabaseName")
    val test= objHiveContext.sql("show tables")
    var i  =0

    var testing = test.collect()
      for(i<-0 until testing.length){    
      println(testing(i))
    } //Able to display all tables
   var tableData= objHiveContext.sql("select * from accounts  c")
     println(tableData) //Displaying all columns in table
     var rowData = tableData.collect() // Throwing Error
    println(rowData)

  }
}

尝试各种@SerialVersionUID（43L）等，但无法弄清楚

错误：

15/12/29 12:38:54 INFO deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15/12/29 12:38:54 INFO FileInputFormat: Total input paths to process : 1
[accounts]
[TableName1]
[tc]
15/12/29 12:38:54 INFO ListSinkOperator: 0 finished. closing... 
15/12/29 12:38:54 INFO ListSinkOperator: 0 Close done
15/12/29 12:38:54 INFO ParseDriver: Parsing command: select * from accounts  c
15/12/29 12:38:54 INFO ParseDriver: Parse Completed
15/12/29 12:38:55 INFO deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/12/29 12:38:55 INFO MemoryStore: ensureFreeSpace(371192) called with curMem=0, maxMem=503379394
15/12/29 12:38:55 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 362.5 KB, free 479.7 MB)
15/12/29 12:38:55 INFO MemoryStore: ensureFreeSpace(32445) called with curMem=371192, maxMem=503379394
15/12/29 12:38:55 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 31.7 KB, free 479.7 MB)
15/12/29 12:38:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.16.101.215:64149 (size: 31.7 KB, free: 480.0 MB)
15/12/29 12:38:55 INFO SparkContext: Created broadcast 0 from toString at String.java:2847
SchemaRDD[6] at sql at CountWords.scala:66
== Query Plan ==
== Physical Plan ==
HiveTableScan [id#15L,name#16,number#17,address#18,city#19,state#20,zip#21,createdby#23,createddate#24,updatedby#25,updateddate#26,status#27,, (MetastoreRelation DbName, accounts, Some(c)), None
15/12/29 12:38:56 INFO MemoryStore: ensureFreeSpace(371192) called with curMem=403637, maxMem=503379394
15/12/29 12:38:56 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 362.5 KB, free 479.3 MB)
15/12/29 12:38:56 INFO MemoryStore: ensureFreeSpace(32445) called with curMem=774829, maxMem=503379394
15/12/29 12:38:56 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 31.7 KB, free 479.3 MB)
15/12/29 12:38:56 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.16.101.215:64149 (size: 31.7 KB, free: 480.0 MB)
15/12/29 12:38:56 INFO SparkContext: Created broadcast 1 from first at CountWords.scala:68
15/12/29 12:39:00 INFO FileInputFormat: Total input paths to process : 1
15/12/29 12:39:00 INFO SparkContext: Starting job: first at CountWords.scala:68
15/12/29 12:39:00 INFO DAGScheduler: Got job 0 (first at CountWords.scala:68) with 1 output partitions (allowLocal=false)
15/12/29 12:39:00 INFO DAGScheduler: Final stage: ResultStage 0(first at CountWords.scala:68)
15/12/29 12:39:00 INFO DAGScheduler: Parents of final stage: List()
15/12/29 12:39:00 INFO DAGScheduler: Missing parents: List()
15/12/29 12:39:00 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[11] at first at CountWords.scala:68), which has no missing parents
15/12/29 12:39:00 INFO MemoryStore: ensureFreeSpace(9104) called with curMem=807274, maxMem=503379394
15/12/29 12:39:00 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 8.9 KB, free 479.3 MB)
15/12/29 12:39:00 INFO MemoryStore: ensureFreeSpace(4500) called with curMem=816378, maxMem=503379394
15/12/29 12:39:00 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.4 KB, free 479.3 MB)
15/12/29 12:39:00 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 172.16.101.215:64149 (size: 4.4 KB, free: 480.0 MB)
15/12/29 12:39:00 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
15/12/29 12:39:00 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[11] at first at CountWords.scala:68)
15/12/29 12:39:00 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
15/12/29 12:39:00 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 10.40.10.80, ANY, 1462 bytes)
15/12/29 12:39:02 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.40.10.80:52658 (size: 4.4 KB, free: 265.1 MB)
15/12/29 12:39:03 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.40.10.80): **java.io.InvalidClassException: org.apache.spark.sql.catalyst.expressions.AttributeReference; local class incompatible: stream classdesc serialVersionUID = -1645430348362071262, local class serialVersionUID = -5921417852507808552**
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:621)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1707)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1345)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)

1 个答案:

答案 0 :(得分：0)

您是否使用两种不同版本的Spark（特别是Spark SQL，因为它抱怨@ECHO OFF Set RegQry=HKLM\Hardware\Description\System\CentralProcessor\0 REG.exe Query %RegQry% > checkOS.txt Find /i "x86" < CheckOS.txt > StringCheck.txt If %ERRORLEVEL% == 0 ( Echo "This is 32 Bit Operating system" set ProgFileDir=%ProgramFiles% ) ELSE ( ECHO "This is 64 Bit Operating System" ) set ProgFileDir= %ProgramFiles(x86)% del checkos.txt del stringcheck.txt echo --------------- echo "%ProgFileDir%" echo ---------------- pause ）？

请参阅此answer

相关问题
最新问题