Kafka Streams 0.10.1＆＃34;无法刷新状态存储＆＃34;

时间：2016-12-30 09:49:03

标签： scala apache-kafka-streams

我尝试使用Kafka Streams 0.10.1在Scala中创建一个简单的聚合示例，尽管我似乎失败了一个简单的＆＃34; count＆＃34;聚合（使用Kafka控制台生产者）。有了这样的代码：

val inputStream: KStream[String, String] = builder.stream("inputTopic")

inputStream
  .map(new KeyValueMapper[String, String, KeyValue[String, String]] {
    override def apply(k: String, v: String): KeyValue[String, String] = {
      new KeyValue[String, String](v, v)
    }
  })
  .groupByKey()
  .count(TimeWindows.of(10000L), "count-test-1")
  .toStream()
  .to("outputTopic")

它失败了＆＃34;无法刷新状态存储计数-test-1＆＃34;，我在帖子的末尾包含了完整的堆栈跟踪。另一方面，如果我使用print（）而不是to（），它就像一个魅力，将结果打印到控制台/终端：

[KTABLE-TOSTREAM-0000000013]: [aa@1483089460000] , 1
[KTABLE-TOSTREAM-0000000013]: [bb@1483089460000] , 1
[KTABLE-TOSTREAM-0000000013]: [cc@1483089460000] , 2
[KTABLE-TOSTREAM-0000000013]: [dd@1483089460000] , 3
[KTABLE-TOSTREAM-0000000013]: [ee@1483089460000] , 4

有没有人知道这种行为可能是什么原因？

仅供参考，我使用的操作系统是Windows 10作为主机（也通过IntelliJ运行Scala应用程序）和用于Kafka（在Docker容器中）和生产者/消费者应用程序的Ubuntu 16.04 VM。但是，我可以确认在Ubuntu VM上运行应用程序时也会遇到问题。

非常感谢您的帮助，我们非常感谢任何见解： - ）

完整的堆栈跟踪：

2016-12-30 08:57:43 INFO  StreamThread:573 - stream-thread [StreamThread-1] Committing task 2_0
2016-12-30 08:57:43 ERROR StreamThread:582 - stream-thread [StreamThread-1] Failed to commit StreamTask 2_0 state:
org.apache.kafka.streams.errors.ProcessorStateException: task [2_0] Failed to flush state store count-test-1
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:331)
        at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:275)
        at org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:576)
        at org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:562)
        at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:538)
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:456)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
Caused by: java.lang.ClassCastException: org.apache.kafka.streams.kstream.Windowed cannot be cast to java.lang.String
        at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24)
        at org.apache.kafka.streams.processor.internals.RecordCollector.send(RecordCollector.java:72)
        at org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:72)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:42)
        at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.ForwardingCacheFlushListener.apply(ForwardingCacheFlushListener.java:35)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.maybeForward(CachingWindowStore.java:103)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.access$200(CachingWindowStore.java:34)
        at org.apache.kafka.streams.state.internals.CachingWindowStore$1.apply(CachingWindowStore.java:86)
        at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:117)
        at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:100)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:118)
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:329)
        ... 6 more
2016-12-30 08:57:43 INFO  StreamThread:268 - stream-thread [StreamThread-1] Shutting down
2016-12-30 08:57:43 INFO  StreamThread:358 - stream-thread [StreamThread-1] Committing consumer offsets of task 0_0
2016-12-30 08:57:43 INFO  StreamThread:358 - stream-thread [StreamThread-1] Committing consumer offsets of task 1_0
2016-12-30 08:57:43 INFO  StreamThread:358 - stream-thread [StreamThread-1] Committing consumer offsets of task 2_0
2016-12-30 08:57:43 INFO  StreamThread:751 - stream-thread [StreamThread-1] Closing a task 0_0
2016-12-30 08:57:43 INFO  StreamThread:751 - stream-thread [StreamThread-1] Closing a task 1_0
2016-12-30 08:57:43 INFO  StreamThread:751 - stream-thread [StreamThread-1] Closing a task 2_0
2016-12-30 08:57:43 INFO  StreamThread:368 - stream-thread [StreamThread-1] Flushing state stores of task 0_0
2016-12-30 08:57:43 INFO  StreamThread:368 - stream-thread [StreamThread-1] Flushing state stores of task 1_0
2016-12-30 08:57:43 INFO  StreamThread:368 - stream-thread [StreamThread-1] Flushing state stores of task 2_0
2016-12-30 08:57:43 ERROR StreamThread:330 - stream-thread [StreamThread-1] Failed while executing StreamTask 2_0 duet to flush state:
org.apache.kafka.streams.errors.ProcessorStateException: task [2_0] Failed to flush state store count-test-1
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:331)
        at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(AbstractTask.java:180)
        at org.apache.kafka.streams.processor.internals.StreamThread$4.apply(StreamThread.java:369)
        at org.apache.kafka.streams.processor.internals.StreamThread.performOnAllTasks(StreamThread.java:328)
        at org.apache.kafka.streams.processor.internals.StreamThread.flushAllState(StreamThread.java:365)
        at org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:301)
        at org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:269)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:252)
Caused by: java.lang.ClassCastException: org.apache.kafka.streams.kstream.Windowed cannot be cast to java.lang.String
        at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24)
        at org.apache.kafka.streams.processor.internals.RecordCollector.send(RecordCollector.java:72)
        at org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:72)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:42)
        at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.ForwardingCacheFlushListener.apply(ForwardingCacheFlushListener.java:35)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.maybeForward(CachingWindowStore.java:103)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.access$200(CachingWindowStore.java:34)
        at org.apache.kafka.streams.state.internals.CachingWindowStore$1.apply(CachingWindowStore.java:86)
        at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:117)
        at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:100)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:118)
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:329)
        ... 7 more
2016-12-30 08:57:43 INFO  StreamThread:347 - stream-thread [StreamThread-1] Closing the state manager of task 0_0
2016-12-30 08:57:43 INFO  StreamThread:347 - stream-thread [StreamThread-1] Closing the state manager of task 1_0
2016-12-30 08:57:43 INFO  StreamThread:347 - stream-thread [StreamThread-1] Closing the state manager of task 2_0
2016-12-30 08:57:43 ERROR StreamThread:330 - stream-thread [StreamThread-1] Failed while executing StreamTask 2_0 duet to close state manager:
org.apache.kafka.streams.errors.ProcessorStateException: task [2_0] Failed to close state store count-test-1
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:351)
        at org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(AbstractTask.java:120)
        at org.apache.kafka.streams.processor.internals.StreamThread$2.apply(StreamThread.java:348)
        at org.apache.kafka.streams.processor.internals.StreamThread.performOnAllTasks(StreamThread.java:328)
        at org.apache.kafka.streams.processor.internals.StreamThread.closeAllStateManagers(StreamThread.java:344)
        at org.apache.kafka.streams.processor.internals.StreamThread.shutdownTasksAndState(StreamThread.java:305)
        at org.apache.kafka.streams.processor.internals.StreamThread.shutdown(StreamThread.java:269)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:252)
Caused by: java.lang.ClassCastException: org.apache.kafka.streams.kstream.Windowed cannot be cast to java.lang.String
        at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24)
        at org.apache.kafka.streams.processor.internals.RecordCollector.send(RecordCollector.java:72)
        at org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:72)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:42)
        at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.ForwardingCacheFlushListener.apply(ForwardingCacheFlushListener.java:35)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.maybeForward(CachingWindowStore.java:103)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.access$200(CachingWindowStore.java:34)
        at org.apache.kafka.streams.state.internals.CachingWindowStore$1.apply(CachingWindowStore.java:86)
        at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:117)
        at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:100)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:118)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.close(CachingWindowStore.java:124)
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:349)
        ... 7 more
2016-12-30 08:57:43 INFO  KafkaProducer:685 - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
2016-12-30 08:57:43 INFO  StreamThread:725 - stream-thread [StreamThread-1] Removing all active tasks [[0_0, 1_0, 2_0]]
2016-12-30 08:57:43 INFO  StreamThread:740 - stream-thread [StreamThread-1] Removing all standby tasks [[]]
2016-12-30 08:57:43 INFO  StreamThread:292 - stream-thread [StreamThread-1] Stream thread shutdown complete
Exception in thread "StreamThread-1" org.apache.kafka.streams.errors.ProcessorStateException: task [2_0] Failed to flush state store count-test-1
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:331)
        at org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:275)
        at org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:576)
        at org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:562)
        at org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:538)
        at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:456)
        at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:242)
Caused by: java.lang.ClassCastException: org.apache.kafka.streams.kstream.Windowed cannot be cast to java.lang.String
        at org.apache.kafka.common.serialization.StringSerializer.serialize(StringSerializer.java:24)
        at org.apache.kafka.streams.processor.internals.RecordCollector.send(RecordCollector.java:72)
        at org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:72)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.KStreamMapValues$KStreamMapProcessor.process(KStreamMapValues.java:42)
        at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:82)
        at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:204)
        at org.apache.kafka.streams.kstream.internals.ForwardingCacheFlushListener.apply(ForwardingCacheFlushListener.java:35)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.maybeForward(CachingWindowStore.java:103)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.access$200(CachingWindowStore.java:34)
        at org.apache.kafka.streams.state.internals.CachingWindowStore$1.apply(CachingWindowStore.java:86)
        at org.apache.kafka.streams.state.internals.NamedCache.flush(NamedCache.java:117)
        at org.apache.kafka.streams.state.internals.ThreadCache.flush(ThreadCache.java:100)
        at org.apache.kafka.streams.state.internals.CachingWindowStore.flush(CachingWindowStore.java:118)
        at org.apache.kafka.streams.processor.internals.ProcessorStateManager.flush(ProcessorStateManager.java:329)
        ... 6 more
2016-12-30 08:57:43 INFO  KafkaStreams:237 - Stopped Kafka Stream process

1 个答案:

答案 0 :(得分：5)

DocumentCSS的结果类型不是count(...)而是<String,Long>，因为您使用了窗口聚合。因此，类型为<Windowed<String>,Long>的默认密钥de / serializer失败：

String

您需要在Caused by: java.lang.ClassCastException: org.apache.kafka.streams.kstream.Windowed cannot be cast to java.lang.String 中指定其他密钥取消/序列化工具，或者您需要在to(...)后再添加map()来将密钥类型从toStream()转换为Windowed<String>。

如果使用String，则无效，因为与将结果写入Kafka主题相比，没有发生序列化。

相关问题
最新问题