卡夫卡与烧瓶。 KafkaTimeoutError:60.0秒后无法更新元数据

时间:2019-07-08 11:18:53

标签: python mongodb flask apache-kafka

如何将kafka与烧瓶整合?我已经做到了,但是在发送消息(producer_instance.send(topic_name, key=key_bytes, value=value_bytes))时遇到了错误。它说KafkaTimeoutError: Failed to update metadata after 60.0 secs.。我在这里发布我的代码。

    #views.py
    from app import app
    from db.mongo import Database
    from flask import jsonify
    import json
    from app.kafka import Producer

    @app.route('/')
    def index():
        kafka_producer = None
        results = get_data()
        if len(results) > 0:
            kafka_producer = Producer.connect_kafka_producer()
            Producer.publish_message(kafka_producer, 'raw_country_info', 'raw', str(results).strip())
        if kafka_producer is not None:
            kafka_producer.close()
        return json.dumps(results, default=str)

    def get_data():
        database = Database().get_db()
        results = []
        for doc in database["database"].get_collection('country_info').find().limit(10):
            results.append(doc)
        return results

    #Producer.py
    import json
    from time import sleep
    from bs4 import BeautifulSoup
    from kafka import KafkaConsumer, KafkaProducer

    def connect_kafka_producer():
        _producer = None
        try:
            _producer = KafkaProducer(bootstrap_servers=['localhost:9092'], api_version=(0, 10))
        except Exception as ex:
            print('Exception while connecting Kafka')
            print(str(ex))
        finally:
            return _producer

    def publish_message(producer_instance, topic_name, key, value):
        try:
            key_bytes = bytes(key, encoding='utf-8')
            value_bytes = bytes(value, encoding='utf-8')
            print("---------", key_bytes, value_bytes)
            producer_instance.send(topic_name, key=key_bytes, value=value_bytes)
            producer_instance.flush()
            print('Message published successfully.')
        except Exception as ex:
            print('Exception in publishing message')
            print(str(ex))

0 个答案:

没有答案