Question

我正在尝试一个scala和java组合项目，我有一个scala类，其缩写结构如下

 case class Dl(name:String, length:Int) extends Serializable 

 class DlStruct private(xs:List[Dl]) extends Serializable {
    def this()= this(Nil)

    private def +=(dl:DataLayout): RowSchema =
      new RowSchema(xs :+ dl)

    def appendDl(fieldName:String, fieldLength:Int):DlStruct=
      this += Dl(fieldName,fieldLength)

 }

从Java对象调用上述类以填充DlStruct，完成后，我将类文件写为序列化文件。

当我再次反序列化文件并将其转换回对象时，使用IntelliJ进行工作时，它工作得很好，但是如果我尝试从spark-submit中运行相同的代码，则会引发以下错误：-

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field somepackage.DlStruct.xs of type scala.collection.immutable.List in instance of somepackage.DlStruct.xs
at java.base/java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2205)
at java.base/java.io.ObjectStreamClass$FieldReflector.checkObjectFieldValueTypes(ObjectStreamClass.java:2168)
at java.base/java.io.ObjectStreamClass.checkObjFieldValueTypes(ObjectStreamClass.java:1422)
at java.base/java.io.ObjectInputStream.defaultCheckFieldValues(ObjectInputStream.java:2450)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2357)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2166)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1668)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2434)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2166)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1668)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:482)
at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:440)

常见的Java对象的反序列化也没有问题

反序列化的代码段

 File file = new File(serializedFilePath);
 FileInputStream fin = new FileInputStream(file);
 ObjectInputStream in = new ObjectInputStream(fin);

 infoHolder = (ObjectCarrier) in.readObject(); // <- this line gives error if it has scala object, else runs smoothly

 in.close();
 fileIn.close();

Spark版本-2.4.4 Scala版本-2.12.8 Java- 1.8

Answer 1

我不得不将scala类转换为Java，以使其最终开始在spark-submit中工作，我希望有人为此找到更好的答案。

Scala类的火花提交中的反灭菌问题

1 个答案: