Question

当我使用kafka-flink进行概念验证时，我发现了以下内容：似乎kafka生产者错误可能是由于在flink端完成的工作量而发生的？

以下是更多详情：

我有样本文件，例如样本??。EDR由~700＆＃39; 000行组成，值为＆＃34;实体＆＃34;，＆＃34;值＆＃34;，＆＃34;时间戳＆＃34 ;

我使用以下命令创建kafka主题：

~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --topic gprs

我使用以下命令在主题上加载示例文件：

[13:00] kafka@ubu19: ~/fms
% /home/kafka/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic gprs < ~/sample/sample01.EDR

我有flink方面的工作，每个实体聚合价值，滑动窗口为6小时72小时（aggregationeachsix，aggregationeachsentytwo）。

我做了三个场景：

在没有任何作业运行的情况下加载主题中的文件
在运行aggregationeachsix作业的主题中加载文件
使用aggregationeachsix加载主题中的文件，并在运行

结果是前两个场景正在运行但是对于第三个场景，我在加载文件时在kafka生产者端出现以下错误（并不总是在同一个文件中，它可以是第一个，第二个，第三个甚至是第一个以后的文件）：

    [plenty of lines before this part]
    [2017-08-09 12:56:53,409] ERROR Error when sending message to topic gprs with key: null, value: 35 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
    org.apache.kafka.common.errors.TimeoutException: Expiring 233 record(s) for gprs-0: 1560 ms has passed since last append
    [2017-08-09 12:56:53,409] ERROR Error when sending message to topic gprs with key: null, value: 37 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
    org.apache.kafka.common.errors.TimeoutException: Expiring 233 record(s) for gprs-0: 1560 ms has passed since last append
    [2017-08-09 12:56:53,409] ERROR Error when sending message to topic gprs with key: null, value: 37 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
    org.apache.kafka.common.errors.TimeoutException: Expiring 233 record(s) for gprs-0: 1560 ms has passed since last append
    [2017-08-09 12:56:53,412] WARN Got error produce response with correlation id 1627 on topic-partition gprs-0, retrying (2 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
    [2017-08-09 12:56:53,412] WARN Got error produce response with correlation id 1626 on topic-partition gprs-0, retrying (2 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
    [2017-08-09 12:56:53,412] WARN Got error produce response with correlation id 1625 on topic-partition gprs-0, retrying (2 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
    [2017-08-09 12:56:53,412] WARN Got error produce response with correlation id 1624 on topic-partition gprs-0, retrying (2 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
    [2017-08-09 12:56:53,515] ERROR Error when sending message to topic gprs with key: null, value: 35 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
    org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for gprs-0: 27850 ms has passed since batch creation plus linger time
    [2017-08-09 12:56:53,515] ERROR Error when sending message to topic gprs with key: null, value: 37 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
    org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for gprs-0: 27850 ms has passed since batch creation plus linger time
    [plenty of lines after this part]

我的问题是为什么flink会对kafka制作人有影响，然后我需要更改什么来避免这个错误？

Answer 1

当flink和kafka-producer都在使用它时你看起来正在使你的网络饱和，因此你得到TimeoutExceptions。

kafka-flink：Kafka生产者错误取决于flink工作

1 个答案: