Question

我有5个节点Kafka集群，Zookeeper在3个节点上运行。我的消费者没有从主题中消耗消息。我的消费者应用程序抛出以下错误

        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] TS=2018-05-23 02:00:56,thread=[kafka-coordinator-heartbeat-thread | dcm-onhands],level=ERROR,appName=${cf:application_name},space=${cf:space_name},class=AbstractCoordinator$HeartbeatThread:51,msg=Uncaught exception in thread 'kafka-coordinator-heartbeat-thread | dcm-onhands':
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:528) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] java.lang.OutOfMemoryError: Direct buffer memory
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at java.nio.Bits.reserveMemory(Bits.java:694) ~[?:1.8.0_152]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_152]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_152]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241) ~[?:1.8.0_152]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[?:1.8.0_152]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_152]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:109) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:145) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:93) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:231) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:192) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:469) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.common.network.Selector.poll(Selector.java:398) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:238) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.pollNoWakeup(ConsumerNetworkClient.java:275) ~[kafka-clients-1.0.0.jar!/:?]
        2018-05-22T22:00:56.354-04:00 [APP/PROC/WEB/2] [OUT] at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:934) ~[kafka-clients-1.0.0.jar!/:?]

我的消费者配置如下：

maxpollrecordsconfig = 25000 sessiontimeoutms = 30000

服务器配置 KAFKA_HEAP_OPTS =＆＃34; -Xmx2G -Xms2G＆＃34;

我也与其他消费者面对这个问题

Answer 1

直接内存（Kafka大量使用性能）在堆上分配（因此不受通常的-Xmx参数控制），并且使用以下命令配置最大值： -XX：MaxDirectMemorySize

如果您有兴趣，本文将介绍更多详细信息： https://dzone.com/articles/default-hotspot-maximum-direct-memory-size

Spring Kafka - 消费者没有从主题中消耗消息。 java.lang.OutOfMemoryError：直接缓冲内存

1 个答案: