Question

我在RichMapFunctin.Like

的构造函数中使用AtomicLongMap

public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {

private final AtomicLongMap<Object> mObjectAtomicLongMap;

public PathAnalysis()
{
    mObjectAtomicLongMap = AtomicLongMap.create();
}
}

注册自定义serizlize类，但不起作用

env.getConfig().registerTypeWithKryoSerializer(AtomicLongMap.class, new AtomicLongMapSerializer());

它导致：

org.apache.flink.api.common.InvalidProgramException: The implementation of the RichMapFunction is not serializable. The object probably contains or references non serializable fields.
    at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:99)
    at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1550)
    at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:184)
    at org.apache.flink.streaming.api.datastream.DataStream.map(DataStream.java:528)
    at com.ghzs.Topology.main(Topology.java:91)
Caused by: java.io.NotSerializableException: org.apache.flink.shaded.curator.org.apache.curator.shaded.com.google.common.util.concurrent.AtomicLongMap
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:315)
    at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:81)
    ... 4 more

AtomicLongMap不是Serializable的实现。如何注册有效的自定义序列化方法？

Answer 1

将AtomicLongMap标记为瞬态，并将其分配给您的函数将接收的open（）调用（因为它是RichMapFunction）。如下所示：

public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {

  private transient AtomicLongMap<Object> mObjectAtomicLongMap;

  public PathAnalysis() { }

  @Override
  public void open(Configuration parameters) throws Exception {
    super.open(parameters);

    mObjectAtomicLongMap = AtomicLongMap.create();
  }
}

Answer 2

尝试执行此功能，如：

    import org.apache.flink.api.common.functions.RichMapFunction;
    import com.google.common.util.concurrent.AtomicLongMap;
    public class PathAnalysis extends RichMapFunction<ApiLog, ApiLog> {
    private final AtomicLongMap<Object> mObjectAtomicLongMap;

    public PathAnalysis()
    {
      mObjectAtomicLongMap = AtomicLongMap.create();
    }

    @Override
    public ApiLog map(ApiLog value) throws Exception {
   // TODO Auto-generated method stub
    return null;
    }
    }

如何在Flink中使用自定义序列化方法？

2 个答案: