Question

我在使用Python机制化循环下载多个文件时遇到问题。我也在使用Beautiful Soup 4.任何一个软件包的文档似乎都没有答案。

这是我的代码 - 请跳到实际的循环。我把所有内容都包括在内作为参考：

import mechanize, cookielib, os, time
from bs4 import BeautifulSoup


fcList = ['abandoned mine land inventory points', 'abandoned mine land inventory polygons', \
          'abandoned mine land inventory sites', 'coal mining operations', 'coal pillar location-mining', \
          'industrial mineral mining operations', 'longwall mining panels', 'mine drainage treatment/land recycling project locations', \
          'mined out areas', 'residual waste operations', 'underground mining permit']

dlLink = 'FTP Download'
dloadPath = 'C:\\Users\\SomeGuy\\Downloads'

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Select the first (index zero) form
br.select_form(nr=0)

# Input form data
br.form['Keyword']='mining'
br.submit()
html = br.response().read()

# Pass html to beautiful soup for parse
soup = BeautifulSoup(html)
htmlinks = soup.findAll("a")

# Find links with desired text
for htmlink in htmlinks:
    string = str(htmlink.string)
    if string.lower() in fcList:
        print "Matched link!", string + ". attempting download...\n"
        try:
            req = br.click_link(text = string)
            br.open(req)
            print "URL: " + str(br.geturl)
            html = br.response().read()
            soup = BeautifulSoup(html)
            the_tag = soup.find('a', text=dlLink)
            fileURL = the_tag.get('href')
            print fileURL
            # attempt download
            fnam = string.replace(" ", "_")
            fnam = fnam.replace("/", "_")
            f = br.retrieve(fileURL, os.path.join(dloadPath, fnam + ".zip"))
            print f + "\n"
            br.back()
        except:
            print "An unknown error occurred."

输出：

>>> 
Matched link! Abandoned Mine Land Inventory Points. attempting download...

URL: <bound method Browser.geturl of <mechanize._mechanize.Browser instance at 0x02D9D7B0>>
http://www.pasda.psu.edu/data/dep/AMLInventoryPoints2013_04.zip
An unknown error occurred.
Matched link! Abandoned Mine Land Inventory Polygons. attempting download...

An unknown error occurred.
Matched link! Abandoned Mine Land Inventory Sites. attempting download...

An unknown error occurred.
Matched link! Coal Mining Operations. attempting download...

An unknown error occurred.
Matched link! Coal Pillar Location-Mining. attempting download...

An unknown error occurred.
Matched link! Industrial Mineral Mining Operations. attempting download...

An unknown error occurred.
Matched link! Longwall Mining Panels. attempting download...

An unknown error occurred.
Matched link! Mine Drainage Treatment/Land Recycling Project Locations. attempting     download...

An unknown error occurred.
Matched link! Mined Out Areas. attempting download...

An unknown error occurred.
Matched link! Residual Waste Operations. attempting download...

An unknown error occurred.
Matched link! Underground Mining Permit. attempting download...

An unknown error occurred.
>>>

我认为问题可能是由于下载之间没有等待时间。无论我选择哪一个，这段代码都会成功下载循环中的第一个文件。或许它是我不知道的其他一些错误 - 我昨天刚刚下载了机械化和美丽的汤！

Answer 1

试试这个：

f = br.retrieve(fileURL, os.path.join(dloadPath, fnam + ".zip"))[0]

如果这不起作用，请删除try..catch并发布您获得的实际错误

使用Python机械化在循环中下载文件

1 个答案: