Question

我非常不熟悉，昨天才发现它，并且只有基本的python经验。

我需要映射每个内部和外部链接的一组子域（大约200个）。

我只是不了解我认为的事情的输出方面。

这是我到目前为止所拥有的。

import scrapy

class LinkSpider(scrapy.Spider):
    name = 'links'

    allowed_domains = ['example.com']
    start_urls = ['https://www.example.com/']

    def parse(self, response):
        # follow all links
        for href in response.css('a::attr(href)'):
            yield response.follow(href, self.parse)

它像这样输出到终端：

DEBUG: Crawled (200) <GET http://www.example.com/> (referer: None)
DEBUG: Crawled (200) <GET http://www.example.com/aaa/A-content-page> (referer: http://www.example.com/)
DEBUG: Crawled (200) <GET http://aaa.example.com/bbb/something/> (referer: http://www.example.com/)

我想要的是CSV或TSV

URL                                         Referer
http://www.example.com/                     None
http://www.example.com/aaa/A-content-page   http://www.example.com/
http://aaa.example.com/bbb/something/       http://www.example.com/
http://aaa.example.com/bbb/another/         http://aaa.example.com/bbb/something/

我们非常感谢您的协助，但希望将其推荐给文档而不是直接的解决方案。

这是我想出的解决方案。

    def parse(self, response):
        filename = "output.tsv"
        f = open(filename, 'w')
        f.write("URL\tLink\tReferer\n")
        f.close()
        # follow all links
        for href in response.css('a::attr(href)'):            
            yield response.follow(href, self.parse)
            with open(filename, 'a') as f:
                url = response.url
                links = response.css('a::attr(href)').getall()
                referer = response.request.headers.get('referer', None).decode('utf-8')
                for item in links:
                    f.write("{0}\t{1}\t{2}\n".format(url, item, referer))

        f.close()

Answer 1

虽然这不是100％正确的，但这是一个很好的开始。

    def parse(self, response):
    filename = "output.tsv"
    # follow all links
    for href in response.css('a::attr(href)'):            
        yield response.follow(href, self.parse)
        with open(filename, 'a') as f:
            links = response.css('a::attr(href)').getall()
            referer = response.request.headers.get('Referer', None)
            for item in links:
                f.write("{0}\t{1}\n".format(item, referer))

Answer 2

您只需在解析中即可获得两个网址。

lp() { php "$HOME/path/to/longProgram.php" "$@"; } referer = response.request.headers.get('Referer')

original_url = response.url

您可以使用

将输出写入文件

yield {'referer': referer, 'url': original_url}

Scrapy搜寻器在搜寻时仅返回URL和Referrer

2 个答案: