Question

我有一个简单的管道

env.addSource(kafkaConsumer).uid("kafka-src").name(consumerName)
    .keyBy(_.id)
    .process(new Processor).uid("processor")
    .addSink(kafkaProducer).name(producerName)

现在我尝试将uid像这样简单地添加到接收器中

env.addSource(kafkaConsumer).uid("kafka-src").name(consumerName)
    .keyBy(_.id)
    .process(new Processor).uid("processor")
    .addSink(kafkaProducer).name(producerName).uid("kafka-sink")

但是我得到了一个很长的例外，似乎这是消息的一部分：

Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint file:/tmp/rocksdb/savepoint-445173-011657873d74. Cannot map checkpoint/savepoint state for operator 3cfeb06db0484d5556a7de8db2025f09 to the new program, because the operator is not available in the new program. If you want to allow to skip this, you can set the --allowNonRestoredState option on the CLI.
    at org.apache.flink.runtime.checkpoint.Checkpoints.loadAndValidateCheckpoint(Checkpoints.java:205)
    at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.restoreSavepoint(CheckpointCoordinator.java:1103)
    at org.apache.flink.runtime.jobmaster.JobMaster.tryRestoreExecutionGraphFromSavepoint(JobMaster.java:1251)
    at org.apache.flink.runtime.jobmaster.JobMaster.createAndRestoreExecutionGraph(JobMaster.java:1175)
    at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:299)
    at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:83)
    at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:37)
    at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:146)

这有意义吗？反正有没有解决它而不丢失保存点？

Answer 1

这个问题很有意义，因为如果您不手动指定ID，它们将自动生成。生成的ID可能是3cfeb06db0484d5556a7de8db2025f09。

您有三个选择：

在没有保存点的情况下开始作业：这将导致数据丢失；
运行允许非还原状态的作业；
将3cfeb06db0484d5556a7de8db2025f09用作您的运营商的uid。

以下一些链接可能会对您有所帮助：

向操作员添加uid导致“无法为操作员映射检查点/保存点状态”

1 个答案: