循环遍历网址列表,运行BeautifulSoup,写入文件

时间:2017-03-08 16:19:06

标签: python for-loop writefile

我有一个我想要运行的网址列表,使用BeautifulSoup清理并保存到.txt文件。

这是我的代码,目前只有列表中的几个项目,还有更多来自txt文件,但是现在这样做很简单。

当循环正在运行时,它会将两个URL的输出传递给URL.txt文件。我希望列表中的每个实例都输出到其唯一的.txt文件。

import urllib
from bs4 import BeautifulSoup


x = ["https://www.sec.gov/Archives/edgar/data/1000298/0001047469-13-002555.txt",
"https://www.sec.gov/Archives/edgar/data/1001082/0001104659-13-011967.txt"]

for url in x:

    #I want to open the URL listed in my list

    fp = urllib.request.urlopen(url)
    test = fp.read()
    soup = BeautifulSoup(test,"lxml")
    output=soup.get_text()

    #and then save the get_text() results to a unique file.

    file=open("url.txt","w",encoding='utf-8')
    file.write(output)
    file.close()

谢谢你看看。最好,乔治

1 个答案:

答案 0 :(得分:0)

为列表中的每个项目创建不同的文件名,如下所示:

import urllib
from bs4 import BeautifulSoup


x = ["https://www.sec.gov/Archives/edgar/data/1000298/0001047469-13-002555.txt",
"https://www.sec.gov/Archives/edgar/data/1001082/0001104659-13-011967.txt"]

for index , url in enumerate(x):

    #I want to open the URL listed in my list

    fp = urllib.request.urlopen(url)
    test = fp.read()
    soup = BeautifulSoup(test,"lxml")
    output=soup.get_text()

    #and then save the get_text() results to a unique file.

    file=open("url%s.txt" % index,"w",encoding='utf-8')
    file.write(output)
    file.close()