How to execute a Scrapy request immediately and get the returned response in parse method?

Question

How to execute a Scrapy request immediately and get the returned response in parse method?

I need to get some information from another link and combine it with the current information I have from the original link (in parse method). Therefore, I need to be able to execute a scrapy request and get the returned response.

Answer 1

我认为你正在寻找ScrapyRT（scrapy real time）

https://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-an-api-for-scrapy-spiders/

参见示例http://localhost:9080/crawl.json?spider_name=foo&url=http://example.com/product/1

  

ScrapyRT将在Scrapy中为指定的URL安排一个请求，并使用'foo'spider的parse方法作为回调。从页面提取的数据将被序列化为JSON并在响应正文中返回。