Question

根据我在网上和其他堆栈溢出帖子上发现的内容，似乎主要有两种方法可以删除kafka上的主题。第一个是：a）delete.topic.enable = true并且正在运行./kafka-topics.sh ---delete --topic <topicName> 第二种方式：./zookeeper-shell.sh localhost:2181 rmr brokers/topics

我注意到第一种方法标记了要删除的每个主题，并且在几分钟内主题被删除，而第二种方法会立即删除它们。我也注意到重启服务器需要几个小时，这是正常的吗？我在一个经纪人身上有超过1000个主题（用于测试目的）。

Answer 1

第一种方法将在zookeper admin/delete_topics/<topic>中创建一个节点，如果你已经像你一样启用了主题删除，那么监视delete_topics子节点的kafka代理（TopicDeletionManager）中的给定线程将会处理这个，这意味着从zookeper中删除，但也从所有kafka副本中删除日志，确保您不会以无效状态结束。整个过程在这里描述： https://github.com/apache/kafka/blob/0.11.0/core/src/main/scala/kafka/controller/TopicDeletionManager.scala

/**
 * This manages the state machine for topic deletion.
 * 1. TopicCommand issues topic deletion by creating a new admin path /admin/delete_topics/<topic>
 * 2. The controller listens for child changes on /admin/delete_topic and starts topic deletion for the respective topics
 * 3. The controller's ControllerEventThread handles topic deletion. A topic will be ineligible
 *    for deletion in the following scenarios -
  *   3.1 broker hosting one of the replicas for that topic goes down
  *   3.2 partition reassignment for partitions of that topic is in progress
 * 4. Topic deletion is resumed when -
 *    4.1 broker hosting one of the replicas for that topic is started
 *    4.2 partition reassignment for partitions of that topic completes
 * 5. Every replica for a topic being deleted is in either of the 3 states -
 *    5.1 TopicDeletionStarted Replica enters TopicDeletionStarted phase when onPartitionDeletion is invoked.
 *        This happens when the child change watch for /admin/delete_topics fires on the controller. As part of this state
 *        change, the controller sends StopReplicaRequests to all replicas. It registers a callback for the
 *        StopReplicaResponse when deletePartition=true thereby invoking a callback when a response for delete replica
 *        is received from every replica)
 *    5.2 TopicDeletionSuccessful moves replicas from
 *        TopicDeletionStarted->TopicDeletionSuccessful depending on the error codes in StopReplicaResponse
 *    5.3 TopicDeletionFailed moves replicas from
 *        TopicDeletionStarted->TopicDeletionFailed depending on the error codes in StopReplicaResponse.
 *        In general, if a broker dies and if it hosted replicas for topics being deleted, the controller marks the
 *        respective replicas in TopicDeletionFailed state in the onBrokerFailure callback. The reason is that if a
 *        broker fails before the request is sent and after the replica is in TopicDeletionStarted state,
 *        it is possible that the replica will mistakenly remain in TopicDeletionStarted state and topic deletion
 *        will not be retried when the broker comes back up.
 * 6. A topic is marked successfully deleted only if all replicas are in TopicDeletionSuccessful
 *    state. Topic deletion teardown mode deletes all topic state from the controllerContext
 *    as well as from zookeeper. This is the only time the /brokers/topics/<topic> path gets deleted. On the other hand,
 *    if no replica is in TopicDeletionStarted state and at least one replica is in TopicDeletionFailed state, then
 *    it marks the topic for deletion retry.

直接从zookeeper删除只是意味着从协调器中删除。当请求元数据时，主题不再存在（好吧，也许它们可能来自缓存），但日志文件不会被删除（至少现在不行，我假设经纪人会检测到日志无效并删除它们在一段时间），但你可能对经纪人有一些不连贯性（如果你处于重新平衡的中间，你可能会破坏很多东西）。这可能意味着某些经纪人会认为它被删除了，而另一些经纪人认为它仍然存在......远非理想。

从目前开始删除fom zookeeper（以及来自经纪人的日志）似乎确实可行，但要注意，它可能会导致冲突，无效状态，随机错误，并且在将来的版本中可能根本不起作用。

使用zookeeper-shall.sh rmr代理/主题删除主题和删除Kafka10上kafka-topics.sh上的标志之间的区别

1 个答案: