Question

好吧，我正在尝试在python中使用Kafka-python软件包（1.3.2）来从我的生产者转移到消费者的简单数据。

制片：

from kafka import KafkaConsumer

consumer = KafkaConsumer('my-topic',
                     group_id='my-group',
                     bootstrap_servers=['localhost:9092'],fetch_min_bytes=1)
for message in consumer:
print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
                                      message.offset, message.key,
                                      message.value))

consumer = KafkaConsumer()
consumer.subscribe(["my-topic"])

消费者：

my-topic:0:5056: key=None value=b'message'
 my-topic:0:5057: key=None value=b'message'

我收到以下消费者的信息：

Error in atexit._run_exitfuncs: Traceback (most recent call last): File "C:\Users\VNK736\AppData\Local\Programs\Python\Python36-32\lib\site-packages\kafka\producer\kafka.py", line 364, in wrapper _self.close() File "C:\Users\VNK736\AppData\Local\Programs\Python\Python36-32\lib\site-packages\kafka\producer\kafka.py", line 420, in close self._sender.join(timeout) File "C:\Users\VNK736\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 1060, in join self._wait_for_tstate_lock(timeout=max(timeout, 0)) File "C:\Users\VNK736\AppData\Local\Programs\Python\Python36-32\lib\threading.py", line 1072, in _wait_for_tstate_lock elif lock.acquire(block, timeout): OverflowError: timeout value is too large

但同时我在制作人处有这个错误：

NONE

默认情况下，超时设置为999999999，并在Kafka.py中设置为If (Directory.Exists(filePath)) Then 'search file in the input path by their search pattern For Each File As String In Directory.GetFiles(filePath, "*.txt", SearchOption.TopDirectoryOnly) Console.WriteLine("Reading the current file " + Path.GetFileName(File)) Using sr As StreamReader = New StreamReader(File) Dim Currentline As String Dim Identifier As String Dim statementDate As String Dim currenttext As String 'getting the unique identifier from the files and removing the white spaces Identifier = sr.ReadLine.Substring(69, 8) 'checks until the EOF While Not sr.EndOfStream currenttext = sr.ReadLine() 'loop through until identified not repeated Do Until currenttext.Contains(Identifier) Currentline = sr.ReadLine() Console.WriteLine(Currentline) Loop Console.WriteLine("=========================== Records Ends") End While End Using。我无法弄清楚在KafkaProducer中传递此超时的参数 - 在我的生产者代码中。

有没有人遇到这个问题？或者任何人都可以帮我这个方向。提前谢谢。

Answer 1

我认为您的问题可能源于您使用的是32位窗口。据我所知，kafka-python代码中没有明确支持32位。

据我所知，你的实现看起来很好。

Answer 2

基于py3文档here，如果超时参数太大（大于TIMEOUT_MAX），则会引发OverflowError。

获取OverflowError：使用kafka-python producer-consumer时超时值太大

2 个答案: