Question

我在Ubuntu 14.04上运行Spark 2.1 我试图在spark中广播一个查找变量。变量的类型为scala.collection.immutable.Map [String，MyObject]

MyObject有以下字段

＆＃39;名称＆＃39; String类型
＆＃39;地址＆＃39; String类型
＆＃39; rangeSet＆＃39; com.google.common.collect。{TreeRangeSet}

线程中的异常＆＃34; main＆＃34; java.io.NotSerializableException：com.google.common.collect.TreeRangeSet

Serialization stack:
    - object not serializable (class: com.google.common.collect.TreeRangeSet, value: {[/101.32.168.0‥/101.32.181.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]})
    - field (class: com.test.MyObject, name: rangeSet, type: class com.google.common.collect.TreeRangeSet)
    - object (class com.test.MyObject, MyObject(Jack,Test,{[/101.32.168.0‥/101.32.181.255][/192.16.10.224‥/192.16.10.255][/4626:7800:4048:0:0:0:0:0‥/4626:7800:4048:ffff:ffff:ffff:ffff:ffff]}))
    - writeObject data (class: scala.collection.immutable.HashMap$SerializationProxy)
    - object (class scala.collection.immutable.HashMap$SerializationProxy, scala.collection.immutable.HashMap$SerializationProxy@708f7386)
    - writeReplace data (class: scala.collection.immutable.HashMap$SerializationProxy)

MyObject.scala

import com.google.common.collect.{TreeRangeSet}
@SerialVersionUID(123L)
case class MyObject(name:String, address:String,rangeSet:TreeRangeSet[CustomInetAddress]) {
}

CustomInetAddress.java

public class CustomInetAddress implements Comparable<CustomInetAddress>, Serializable {

    private InetAddress inetAddress;

    public CustomInetAddress(String ip) throws UnknownHostException {
        this.inetAddress = InetAddress.getByName(ip);
    }

    public CustomInetAddress(InetAddress address) throws UnknownHostException {
        this.inetAddress = address;
    }

    @Override
    public int compareTo(final CustomInetAddress address){
        byte[] ba1 = this.inetAddress.getAddress();
        byte[] ba2 = address.inetAddress.getAddress();

        if(ba1.length < ba2.length) return -1;
        if(ba1.length > ba2.length) return 1;

        for(int i = 0; i < ba1.length; i++) {
            int b1 = unsignedByteToInt(ba1[i]);
            int b2 = unsignedByteToInt(ba2[i]);
            if(b1 == b2)
                continue;
            if(b1 < b2)
                return -1;
            else
                return 1;
        }
        return 0;
    }

    @Override
    public String toString(){
        return this.inetAddress.toString();
    }

    private int unsignedByteToInt(byte b) {
        return (int) b & 0xFF;
    }
}

TreeRangeSet [CustomInetAddress]是对象的实际类型。 CustomInetAddress有一个InetAddress类型的字段。所有这些都是可序列化的。我不确定为什么会抛出异常。

Answer 1

消息很明确：com.google.common.collect.TreeRangeSet没有实现Serializable。

Spark上下文广播变量抛出java.io.NotSerializableException，即使它的Serializable

1 个答案: