Question

我在RocksDB中使用带有增量检查点的Flink 1.4.2，并将检查点保存到S3存储桶中。检查点的结构是一个清单文件，指向包含该状态的某些文件。在文本编辑器中打开清单文件时，我看到一些不可读的块和一些s3 url。

我如何反序列化此清单文件以获得S3 url列表？

Answer 1

https://github.com/king/bravo和https://github.com/sjwiesman/flink/tree/savepoint-connector都包含可以读取和写入保存点/检查点的连接器。您可能会直接或举一个例子，发现其中之一或两者都有用。请参见这张Jira票证https://issues.apache.org/jira/browse/FLINK-12047，以跟踪正在进行的工作，以创建更好的工具来处理Flink快照。

Answer 2

Apache Flink运行时库中的类SavepointStore包含用于存储和加载保存点的方法。

就我当前的情况而言，我创建了此代码片段来检索与检查点相关的文件。

import org.apache.flink.runtime.checkpoint.savepoint.Savepoint;
import org.apache.flink.runtime.checkpoint.savepoint.SavepointStore;
import org.apache.flink.runtime.state.IncrementalKeyedStateHandle;
import org.apache.flink.runtime.state.KeyGroupsStateHandle;
import org.apache.flink.runtime.state.StreamStateHandle;
import org.apache.flink.runtime.state.filesystem.FileStateHandle;
import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;

import java.io.IOException;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class CheckpointFileLocator {

    public static void main(String[] args) throws IOException {
        System.out.println(new CheckpointFileLocator()
                .getS3Locations("/Users/ezequiel/Downloads/chk-3-checkpoint_metadata-f350e54becb2"));
    }

    public Set<String> getS3Locations(String manifestPath) throws IOException {
        Savepoint savepoint = SavepointStore.loadSavepoint(manifestPath, this.getClass().getClassLoader());

        Stream<String> rawStream = savepoint.getOperatorStates().stream()
                .flatMap(operatorState -> operatorState.getSubtaskStates().values().stream())
                .flatMap(operatorSubtaskState -> operatorSubtaskState.getRawKeyedState().stream())
                .map(keyedStateHandle -> (KeyGroupsStateHandle) keyedStateHandle)
                .map(KeyGroupsStateHandle::getDelegateStateHandle)
                .map(this::getPath);

        Stream<String> metadataStream = savepoint.getOperatorStates().stream()
                .flatMap(operatorState -> operatorState.getSubtaskStates().values().stream())
                .flatMap(operatorSubtaskState -> operatorSubtaskState.getManagedKeyedState().stream())
                .map(keyedStateHandle -> (IncrementalKeyedStateHandle) keyedStateHandle)
                .map(IncrementalKeyedStateHandle::getMetaStateHandle)
                .map(this::getPath);

        return Stream.concat(rawStream, metadataStream).collect(Collectors.toSet());
    }

    private String getPath(StreamStateHandle streamStateHandle) {
        if (streamStateHandle instanceof FileStateHandle) {
            return ((FileStateHandle) streamStateHandle).getFilePath().toString();
        } else if (streamStateHandle instanceof ByteStreamStateHandle) {
            return ((ByteStreamStateHandle) streamStateHandle).getHandleName();
        }
        return null;
    }

}

如何反序列化Flink中的外部检查点清单？

2 个答案: