Kafka connect throttling

Question

I have a requirement to consume messages on behalf of a set of lazy consumers who just exposes REST APIs. Therefore, I am planning to have Sink Connectors which fetches messages from Kafka topics and does HTTP POST operation on the exposed APIs.

One of the key factors for consideration is throttling. What mechanism do you suggest for throttling the Sink Tasks to meet the tier SLA of the APIs. I understand that Kafka has client quota feature, however, what is the optimum mechanism to keep track of API requests/min or sec which would allow to adjust the client quota dynamically ?

Answer 1

我认为，对于REST API实施速率限制的最佳方法是在SinkTask.put()中根据需要阻止您的连接器代码。您可能想要考虑在SinkTask级别上的速率限制是否足够，或者您是否需要全局（由于涉及协调而更复杂）。

您正在考虑使用Kafka配额的优点是分布式方面是为您处理的，但我相信目前只能根据传输的字节配置它们。