Question

您好我在我的应用程序中使用kafka进行排队。我正在向我的应用程序提取75k记录/秒，这将由kafka排队。我的应用程序部署在openstack vms上。由于某些基础设施问题，当kafka将这些记录存储到磁盘上时，我面临与记录损坏有关的CRC问题。以下是例外情况：

org.apache.kafka.common.KafkaException: Error deserializing key/value for partition TcpMessage-3 at offset 1331363158
        at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:628) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.Fetcher.handleFetchResponse(Fetcher.java:566) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.Fetcher.access$000(Fetcher.java:69) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:139) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.Fetcher$1.onSuccess(Fetcher.java:136) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:380) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:274) [kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) [kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) [kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) [kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908) [kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853) [kafka-clients-0.9.0.1.jar:?]
        at com.affirmed.mediation.edr.kafka.tcpMessage.TcpMessageConsumer.doWork(TcpMessageConsumer.java:196) [EdrServer.jar:?]
        at com.affirmed.mediation.edr.kafka.tcpMessage.TcpMessageConsumer.run(TcpMessageConsumer.java:255) [EdrServer.jar:?]
Caused by: org.apache.kafka.common.record.InvalidRecordException: **Record is corrupt (stored crc = 2053731240, computed crc = 2767221639)**
        at org.apache.kafka.common.record.Record.ensureValid(Record.java:226) ~[kafka-clients-0.9.0.1.jar:?]
        at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:617) ~[kafka-clients-0.9.0.1.jar:?]
        ... 15 more

那么有没有办法将kafka用于排队而不将记录存储到磁盘上？如果是的话，我们怎样才能实现呢？

Answer 1

那么有没有办法将kafka用于排队而不将记录存储到磁盘上？如果是的话，我们怎样才能实现呢？

一般来说，不，这是不可能的。

你可能做的（粗略！）解决方法是使用RAM drive并配置你的Kafka经纪人将他们的数据存储在这个RAM驱动器上。当然，使用RAM驱动器有几个缺点，例如存在很大的数据丢失风险，因为数据不能持久存储。它还假设您的OpenStack VM的内存不会遭受与其磁盘相同的损坏问题。

也许更好的方法是修复您的OpenStack环境......？

我们可以将Kafka用于排队目的而无需存储到磁盘上吗？如果是这样，我们怎样才能实现呢？

1 个答案: