现有CDH 5.5.2集群上的Kafka配置

Question

我在现有的CDH 5.5.2集群上安装Kafka-2.0，这是我遵循的程序

1. 从CM
添加服务
2. 选择Kafka（之前我在所有节点上下载并分发并激活了kafka parcel）
3. 为KafkaBroker选择1个节点，为Kafka MirrorMaker选择4个节点
4. 然后我用一个Mirror Maker节点以及具有相同节点的Source Broker List（source.bootstrap.servers）更新了我的目标代理列表（bootstrap.servers）属性

5. 我收到的错误（日志文件）

   Fatal error during KafkaServerStartable startup. Prepare to shutdown 
   java.lang.OutOfMemoryError: Java heap space
       at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
       at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
       at kafka.log.SkimpyOffsetMap.<init>(OffsetMap.scala:43)
       at kafka.log.LogCleaner$CleanerThread.<init>(LogCleaner.scala:186)
       at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
       at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
       at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
       at scala.collection.immutable.Range.foreach(Range.scala:166)
       at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
       at scala.collection.AbstractTraversable.map(Traversable.scala:104)
       at kafka.log.LogCleaner.<init>(LogCleaner.scala:83)
       at kafka.log.LogManager.<init>(LogManager.scala:64)
       at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:601)
       at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
       at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
       at kafka.Kafka$.main(Kafka.scala:67)
       at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
       at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
   

Answer 1

您需要将broker_max_heap_size值增加至至少 1GB ，然后从Cloudera Manager重新启动kafka服务。如果您仍面临同样的问题，请尝试根据群集配置进行增加

Answer 2

堆栈跟踪显示＆＃34; java.lang.OutOfMemoryError：Java堆空间＆＃34; - VM堆空间不足。通过设置

来增加它

export KAFKA_HEAP_OPTS="-Xmx1G -Xms512M" 

在/bin/kafka-server-start.sh。