我有一个程序,它从 RabbitMQ 的队列 获取输入,然后运行一个传递此输入的爬虫蜘蛛。它工作正常,但只是第一次,在第一次输入后它会抛出这个错误:
twisted.internet.error.ReactorNotRestartable
这是我的代码:
def start_flow(crawler_class):
# Create the queue and all the connections
queue = Queue(connection_uri='amqp://guest:guest@localhost:5672/')
queue.create_connection()
queue.create_channel()
queue_name = _get_queue_name(crawler_class)
queue.define_queue(queue_name)
# ---------------------------------------
while True:
print("Waiting for the next input")
# Get an input
queue.consume(queue_name)
queue.channel.start_consuming()
input_ = queue.get_next_input()
# ------------
# Runs the crawler passing the input
runner = CrawlerRunner()
d = runner.crawl(crawler_class, input=input_)
d.addCallback(lambda _: reactor.stop())
reactor.run()
# ----------------------------
我已经尝试过 CrawlerProcess,但它抛出了同样的问题