对于Kafka来说，它相对较新，在实现Kafka Producer时遇到了麻烦，在该生产器中，它从函数接收输入message并进一步在定义的主题上进行生产。但是困难在于，将其发布到该主题上需要花费大量时间。而且，如果我尝试更改其配置，它会跳过一些消息，或者会降低在该主题上生成消息的速度。我将在下面发布我的生产者代码，如果有人可以帮助我解决我的问题，我将不胜感激。
代码：-
int rdkafka_produce (json_object *message) {
rd_kafka_t *rk; /* Producer instance handle */
rd_kafka_topic_t *rkt; /* Topic object */
rd_kafka_conf_t *conf; /* Temporary configuration object */
char errstr[512]; /* librdkafka API error reporting buffer */
char buf[2048]; /* Message value temporary buffer */
const char *brokers; /* Argument: broker list */
const char *topic; /* Argument: topic to produce to */
int sendcnt=0;
int partition = RD_KAFKA_PARTITION_UA;
rd_kafka_topic_conf_t *topic_conf;
rd_kafka_resp_err_t err;
rd_kafka_headers_t *hdrs = NULL;
/*
* Argument validation
*/
brokers = "localhost:9092";
topic = "tt_stream";
/*
* Create Kafka client configuration place-holder
*/
conf = rd_kafka_conf_new();
/* Set bootstrap broker(s) as a comma-separated list of
* host or host:port (default port 9092).
* librdkafka will use the bootstrap brokers to acquire the full
* set of brokers from the cluster. */
if (rd_kafka_conf_set(conf, "bootstrap.servers", brokers,
errstr, sizeof(errstr)) != RD_KAFKA_CONF_OK) {
fprintf(stderr, "%s\n", errstr);
return 1;
}
/* Set the delivery report callback.
* This callback will be called once per message to inform
* the application if delivery succeeded or failed.
* See dr_msg_cb() above. */
rd_kafka_conf_set_dr_msg_cb(conf, dr_msg_cb);
if (!(rk = rd_kafka_new(RD_KAFKA_PRODUCER, conf,
errstr, sizeof(errstr)))) {
fprintf(stderr,
"%% Failed to create new producer: %s\n",
errstr);
exit(1);
}
/* Add brokers */
if (rd_kafka_brokers_add(rk, brokers) == 0) {
fprintf(stderr, "%% No valid brokers specified\n");
/* Set the delivery report callback.
* This callback will be called once per message to inform
* the application if delivery succeeded or failed.
* See dr_msg_cb() above. */
rd_kafka_conf_set_dr_msg_cb(conf, dr_msg_cb);
if (!(rk = rd_kafka_new(RD_KAFKA_PRODUCER, conf,
errstr, sizeof(errstr)))) {
fprintf(stderr,
"%% Failed to create new producer: %s\n",
errstr);
exit(1);
}
/* Add brokers */
if (rd_kafka_brokers_add(rk, brokers) == 0) {
fprintf(stderr, "%% No valid brokers specified\n");
exit(1);
}
/* Create topic */
rkt = rd_kafka_topic_new(rk, topic, topic_conf);
topic_conf = NULL; /* Now owned by topic */
//while (run && strcpy(buf, json_object_to_json_string(message))) {
strcpy(buf, json_object_to_json_string(message));
size_t len = strlen(buf);
if (buf[len-1] == '\n')
buf[--len] = '\0';
err = RD_KAFKA_RESP_ERR_NO_ERROR;
/* Send/Produce message. */
if (hdrs) {
rd_kafka_headers_t *hdrs_copy;
hdrs_copy = rd_kafka_headers_copy(hdrs);
err = rd_kafka_producev(
rk,
RD_KAFKA_V_RKT(rkt),
RD_KAFKA_V_PARTITION(partition),
RD_KAFKA_V_MSGFLAGS(RD_KAFKA_MSG_F_COPY),
RD_KAFKA_V_VALUE(buf, len),
RD_KAFKA_V_HEADERS(hdrs_copy),
RD_KAFKA_V_END);
if (err)
rd_kafka_headers_destroy(hdrs_copy);
} else {
if (rd_kafka_produce(
rkt, partition,
RD_KAFKA_MSG_F_COPY,
/* Payload and length */
buf, len,
/* Optional key and its length */
NULL, 0,
/* Message opaque, provided in
* delivery report callback as
* msg_opaque. */
NULL) == -1) {
err = rd_kafka_last_error();
}
}
if (err) {
fprintf(stderr,
"%% Failed to produce to topic %s "
"partition %i: %s\n",
rd_kafka_topic_name(rkt), partition,
rd_kafka_err2str(err));
/* Poll to handle delivery reports */
rd_kafka_poll(rk, 0);
}
sendcnt++;
/* Poll to handle delivery reports */
rd_kafka_poll(rk, 0);
//}
/* Poll to handle delivery reports */
rd_kafka_poll(rk, 0);
/* Wait for messages to be delivered */
while (run && rd_kafka_outq_len(rk) > 0)
rd_kafka_poll(rk, 100); //This is where most of the time is being spent.
/* Destroy topic */
rd_kafka_topic_destroy(rkt);
/* Destroy the handle */
rd_kafka_destroy(rk);
return 0;
}
答案 0 :(得分：1)
您正在为您产生的每条消息创建一个新的生产者客户实例； 这是非常昂贵的，因为它需要旋转线程，连接到引导代理，进行身份验证，执行元数据查找，连接到适当的代理等，然后才能生成单个消息。
相反，您可以为每个消息重用一个长期存在的生产者实例，这会将您的延迟减少到几毫秒（取决于代理连接和负载）。