带状态存储的流应用程序最多需要1个小时才能重新启动

时间：2019-03-08 09:01:01

标签： apache-kafka apache-kafka-streams spring-cloud-stream rocksdb

我们正在使用带有Kafka 2.0.1的春季云流，并利用InteractiveQueryService从商店中获取数据。有4个存储在聚合数据后将数据保留在磁盘上。拓扑的代码如下：

@Slf4j
@EnableBinding(SensorMeasurementBinding.class)
public class Consumer {

  public static final String RETENTION_MS = "retention.ms";
  public static final String CLEANUP_POLICY = "cleanup.policy";

  @Value("${windowstore.retention.ms}")
  private String retention;

/**
 * Process the data flowing in from a Kafka topic. Aggregate the data to:
 * - 2 minute
 * - 15 minutes
 * - one hour
 * - 12 hours
 *
 * @param stream
 */
@StreamListener(SensorMeasurementBinding.ERROR_SCORE_IN)
public void process(KStream<String, SensorMeasurement> stream) {

    Map<String, String> topicConfig = new HashMap<>();
    topicConfig.put(RETENTION_MS, retention);
    topicConfig.put(CLEANUP_POLICY, "delete");

    log.info("Changelog and local window store retention.ms: {} and cleanup.policy: {}",
            topicConfig.get(RETENTION_MS),
            topicConfig.get(CLEANUP_POLICY));

    createWindowStore(LocalStore.TWO_MINUTES_STORE, topicConfig, stream);
    createWindowStore(LocalStore.FIFTEEN_MINUTES_STORE, topicConfig, stream);
    createWindowStore(LocalStore.ONE_HOUR_STORE, topicConfig, stream);
    createWindowStore(LocalStore.TWELVE_HOURS_STORE, topicConfig, stream);
  }

  private void createWindowStore(
        LocalStore localStore,
        Map<String, String> topicConfig,
        KStream<String, SensorMeasurement> stream) {

    // Configure how the statestore should be materialized using the provide storeName
    Materialized<String, ErrorScore, WindowStore<Bytes, byte[]>> materialized = Materialized
            .as(localStore.getStoreName());

    // Set retention of changelog topic
    materialized.withLoggingEnabled(topicConfig);

    // Configure how windows looks like and how long data will be retained in local stores
    TimeWindows configuredTimeWindows = getConfiguredTimeWindows(
            localStore.getTimeUnit(), Long.parseLong(topicConfig.get(RETENTION_MS)));

    // Processing description:
    // The input data are 'samples' with key <installationId>:<assetId>:<modelInstanceId>:<algorithmName>
    // 1. With the map we add the Tag to the key and we extract the error score from the data
    // 2. With the groupByKey we group  the data on the new key
    // 3. With windowedBy we split up the data in time intervals depending on the provided LocalStore enum
    // 4. With reduce we determine the maximum value in the time window
    // 5. Materialized will make it stored in a table
    stream
            .map(getInstallationAssetModelAlgorithmTagKeyMapper())
            .groupByKey()
            .windowedBy(configuredTimeWindows)
            .reduce((aggValue, newValue) -> getMaxErrorScore(aggValue, newValue), materialized);
  }

  private TimeWindows getConfiguredTimeWindows(long windowSizeMs, long retentionMs) {
    TimeWindows timeWindows = TimeWindows.of(windowSizeMs);
    timeWindows.until(retentionMs);
    return timeWindows;
  }

  /**
   * Determine the max error score to keep by looking at the aggregated error signal and
   * freshly consumed error signal
   *
   * @param aggValue
   * @param newValue
   * @return
   */
  private ErrorScore getMaxErrorScore(ErrorScore aggValue, ErrorScore newValue) {
    if(aggValue.getErrorSignal() > newValue.getErrorSignal()) {
        return aggValue;
    }
    return newValue;
  }

  private KeyValueMapper<String, SensorMeasurement,
        KeyValue<? extends String, ? extends ErrorScore>> getInstallationAssetModelAlgorithmTagKeyMapper() {
    return (s, sensorMeasurement) -> new KeyValue<>(s + "::" + sensorMeasurement.getT(),
            new ErrorScore(sensorMeasurement.getTs(), sensorMeasurement.getE(), sensorMeasurement.getO()));
  }
}

因此，在确定特定键的特定窗口内的最大值之后，我们正在将聚合数据具体化到四个不同的存储中。 请注意，保留期设置为两个月的数据，清理策略将删除。我们不会压缩数据。

磁盘上各个状态存储的大小介于14到20 gb之间。

我们正在使用交互式查询：https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html#interactive-queries

在我们的设置中，我们有4个流媒体应用实例用作一个消费者组。因此，每个实例将在其存储中存储所有数据的特定部分。

这一切似乎都很好。直到我们重新启动一个或多个实例并等待它再次可用为止。我希望该应用程序的重启不会花那么长时间，但不幸的是，它最多需要1个小时。我想这个问题是由恢复状态存储的数据量引起的，但是我不确定。我曾期望，当我们将状态存储数据持久保存在运行kubernetes的容器之外的持久卷上时，该应用程序将收到来自代理的最后一个偏移量，并且仅需从该点继续运行，因为以前使用的数据已经存在在州立商店。不幸的是，我不知道如何解决这个问题。

重新启动我们的应用会触发还原任务：

-StreamThread-2] Restoring task 4_3's state store twelve-hours-error-score from beginning of the changelog anomaly-timeline-twelve-hours-error-score-changelog-3.

此过程需要一段时间。为什么要从头开始恢复，为什么要花这么长时间？我确实将auto.offset.reset设置为“最早”，但这仅在不知道偏移量的情况下使用吗？

这是我的信息流设置。注意将max.bytes.buffering设置为0。我更改了此设置，但这没什么区别。我还读到了有关num.stream.threads的错误，其中> 1会引起问题，但是将此错误设置为1并不会提高重新启动速度。

2019-03-05 13:44:53,360 INFO main org.apache.kafka.common.config.AbstractConfig StreamsConfig values: 
application.id = anomaly-timeline
application.server = localhost:5000
bootstrap.servers = [localhost:9095]
buffered.records.per.partition = 1000
cache.max.bytes.buffering = 0
client.id = 
commit.interval.ms = 500
connections.max.idle.ms = 540000
default.deserialization.exception.handler = class org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.key.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
default.production.exception.handler = class org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class errorscore.raw.boundary.ErrorScoreTimestampExtractor
default.value.serde = class errorscore.raw.boundary.ErrorScoreSerde
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
num.standby.replicas = 1
num.stream.threads = 2
partition.grouper = class org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 40000
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
state.cleanup.delay.ms = 600000
state.dir = ./state-store
topology.optimization = none
upgrade.from = null
windowstore.changelog.additional.retention.ms = 86400000

一段时间后，它还会记录以下消息：

CleanupThread] Deleting obsolete state directory 1_1 for task 1_1 as 1188421ms has elapsed (cleanup delay is 600000ms).

另外要注意的是，我确实添加了以下代码，以覆盖在默认情况下删除存储的开始和停止位置的默认清理：

@Bean
public CleanupConfig cleanupConfig() {
    return new CleanupConfig(false, false);
}

任何帮助将不胜感激！

1 个答案:

答案 0 :(得分：1)

我们认为我们已经解决了问题。每个不同的实例都有自己的持久卷。重新启动实例时，似乎某些或有时所有实例已链接到其他持久卷，而不是以前使用的实例。这导致状态存储变得过时，并且恢复过程开始了。我们通过利用NFS共享持久卷来解决此问题，所有实例都指向相同的状态存储目录结构。这似乎解决了问题

相关问题
最新问题