使用python中的scrapy从网站抓取网址和日期

时间:2019-12-16 06:23:31

标签: python scrapy

import scrapy
class QuotesSipder(scrapy.Spider):
    name = "cnbc-news"
    #api_url = "https://www.cnbc.com/search/?query=crude%20oil&qsearchterm=crude%20oil"
    starts_url = ["https://www.cnbc.com/search/?query=crude%20oil&qsearchterm=crude%20oil"]

    def parse(self, response):
        data = json.loads(response.text)
        for link in data['link']:
            yield{
                'link' : 'resultlink'
                'date' : 'SearchResult-publishedDate'
            }

0 个答案:

没有答案