Question

我对此很陌生，我的任务是：让我说出我到底需要什么，我想抓紧搜索并下载某些类型为EX-10.1，EX-10.2等（最高为EX）的合同-10.99。合同可用.htm格式和txt格式。 1.如何取消链接并下载。 2.如何过滤文件类型并下载。 Scrapy希望通过此路径进行下载，路径：url->进入每个CIK链接->搜索并下载EX-10 Type文件。

我的代码

import urlparse

from scrapy.http import Request
from scrapy.spiders import BaseSpider
class legco(BaseSpider):
name = "sec_gov"

allowed_domains = ["www.sec.gov", "search.usa.gov", "secsearch.sec.gov"]
start_urls = ["https://www.sec.gov/cgi-bin/browse-edgar?company=&match=&CIK=&filenum=&State=&Country=&SIC=2834&owner=exclude&Find=Find+Companies&action=getcompany"]


#extract search results
def parse(self, response): 
for link in response.xpath('//div[@id="seriesDiv"]//table[@class="tableFile2"]/a/@href').extract():
    req = Request(url = link, callback = self.parse_page)
    yield req

def parse(self, response):
base_url = 'http://www.sec.gov/cgi-bin/browse-edgar'
for a in response.xpath('//a[@href]/@href'):
    link = a.extract()
    if link.endswith('.htm'):
        link = urlparse.urljoin(base_url, link)
        yield Request(link, callback = self.save_pdf)

def save_pdf(self, response):
path = response.url.split('/')[-1]
with open(path, 'wb') as f:
    f.write(response.body)

我必须做些什么改变？谁能帮我解决这个问题。谢谢你。

抓取抓取并下载特定的Type文件

0 个答案: