Question

我已经用scrapy编写了一些python代码来从网站中提取一些地址。

代码的第一部分是通过读取单独文件googlecoords.txt中的纬度和经度坐标来组合start_urls，然后该文件构成start_urls的一部分。（我之前准备的googlecoords.txt文件在googlemaps的google坐标中转换了英国邮政编码。）

因此，例如，start_url列表中的第一项是“https://www.howdens.com/process/searchLocationsNear.php?lat=53.674434&lon=-1.4908923&distance=1000&units=MILES”，其中“lat = 53.674434＆amp; lon = -1.4908923”来自googlecoors.txt文件。

然而，当我运行代码时，它完美地工作，除了它首先打印出googlecoords.txt文件 - 我不需要。

如何停止此打印？（虽然我可以忍受它。）

import scrapy
import sys

from scrapy.http import FormRequest, Request
from Howdens.items import HowdensItem

class howdensSpider(scrapy.Spider):
    name = "howdens"
    allowed_domains = ["www.howdens.com"]

    # read the file that has a list of google coordinates that are converted from postcodes
    with open("googlecoords.txt") as f:
        googlecoords = [x.strip('\n') for x in f.readlines()]

    # from the goole coordinates build the start URLs
    start_urls = []
    for a in range(len(googlecoords)):
        start_urls.append("https://www.howdens.com/process/searchLocationsNear.php?{}&distance=1000&units=MILES".format(googlecoords[a]))

    # cycle through 6 of the first relevant items returned in the text
    def parse(self, response):
        for sel in response.xpath('/html/body'):
            for i in range(0,6):
                try:
                    item = HowdensItem()
                    item['name'] =sel.xpath('.//text()').re(r'(?<="name":")(.*?)(?=","street")')[i]
                    item['street'] =sel.xpath('.//text()').re(r'(?<="street":")(.*?)(?=","town")')[i]
                    item['town'] = sel.xpath('.//text()').re(r'(?<="town":")(.*?)(?=","pc")')[i]
                    item['pc'] = sel.xpath('.//text()').re(r'(?<="pc":")(.*?)(?=","state")')[i]
                    yield item
                except IndexError:
                    pass

Answer 1

与评论中指出的人一样，您应该使用json方法中的start_requests()模块加载它：

import scrapy
import json

class MySpider(scrapy.Spider):
    start_urls = ['https://www.howdens.com/process/searchLocationsNear.php?lat=53.674434&lon=-1.4908923&distance=1000&units=MILES']

    def parse(self, response):
        data = json.loads(response.body_as_unicode())
        items = data['response']['depots'] 
        for item in items:
            url_template = "https://www.howdens.com/process/searchLocationsNear.php?{}&distance=1000&units=MILES"
            url = url_template.format(item['lat'])  # format in your location here
            yield scrapy.Request(url, self.parse_item)

    def parse_item(self, response): 
        print(response.url)

python scrapy代码打印出我正在从

1 个答案: