Question

我想使用ListCheckpointed将地图存储在我的windowfunciton中（地图是由键共享的，所以我使用了操作员状态），但是重新启动后地图并未从检查点还原。

首先，我使用存储在内存中的Map来触发稍后窗口功能到达时的Map计算。重新启动作业后，地图会丢失，因此我考虑将地图存储在操作员状态中。我已经在环境中设置了检查点，调度了快照状态函数，但是当我取消作业并再次启动它时，地图没有从检查点恢复。

职位代码：


    var config:Configuration = new Configuration()
    config.setBoolean(ConfigConstants.LOCAL_START_WEBSERVER, true)
    val env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(config)
    env.setParallelism(parallelism.toInt)
    env.enableCheckpointing(10 * 1000L)
    env.setStateBackend(new FsStateBackend(checkpointProps("directory")))
    env.getCheckpointConfig.setMinPauseBetweenCheckpoints(500)
    env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
    env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)

    val stream = env
    ...//do watermarks
    .keyBy(a => a.room + "_" + a.role)
 .window(TumblingEventTimeWindows.of(Time.milliseconds(20 * 1000)))
      .apply(new myWindowFunciton())
    env.execute(name)

myWindowFunciton代码：

class myWindowFunciton extends RichWindowFunction[myForm, myForm, String, TimeWindow]
        with ConfigParse with Serializable with Logging  with ListCheckpointed[util.HashMap[String, (myForm, myForm)]] {
    var map2: util.HashMap[String, (myForm, myForm)] = new util.HashMap[String, (myForm, myForm)]()
    ...
    override def apply...
    ...
    override def snapshotState(checkpointId: Long, timestamp: Long): util.List[util.HashMap[String, (myForm, myForm)]] = {
        println("snap map is " + map2.size().toString())
        util.Collections.singletonList(map2)
    }

    override def restoreState(stateList: util.List[util.HashMap[String, (myForm, myForm)]]): Unit = {
        println("restoreState size is " + stateList.size().toString)
        for(stateMap <- stateList.asScala) {
            println("restore every time is " + stateMap.size().toString())
            map2.putAll(stateMap)
        }

        println("after restore:" + map2.size().toString())
    }

}

我在Intellij Idea中进行了测试。我已启动并在快照状态中打印了println，但在webui中取消并重新启动程序后，从未执行过restoreState中的println。

Flink功能扩展的ListCheckpointed不会从检查点恢复

0 个答案: