Question

我正在从txt文件中抓取网址并将其导出到csv文件。但在完成所有过程后，我的代码只会写入最后一个URL的信息。我的猜测是我忘记了一个循环。但是哪里？这是我的代码：

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib import urlopen

file = open('urls.txt', 'r')
filelines = (line.strip() for line in file)
for code in filelines:
    site = urlopen(code)
    soup = BeautifulSoup(site, "html.parser")
    final = soup.find_all("span", {"class": "bd js-title-main-info"})
    print final

records = []
for pagetxt in final:
    print pagetxt.text
    records.append((pagetxt.text))
df = pd.DataFrame(records, columns=['product name'])  
df.to_csv('test.csv', index=False, encoding='utf-8')

由于

Answer 1

从文件中获取数据时，只保留变量final中的最后一个值。尝试提前附加数据（我已使用#####标记了更改）：

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib import urlopen

file = open('urls.txt', 'r')
filelines = (line.strip() for line in file)
records = []                         ######
for code in filelines:
    site = urlopen(code)
    soup = BeautifulSoup(site, "html.parser")
    final = soup.find_all("span", {"class": "bd js-title-main-info"})
    print final

    for pagetxt in final:               ######
       print pagetxt.text               ######
       records.append((pagetxt.text))   ######   

df = pd.DataFrame(records, columns=['product name'])  
df.to_csv('test.csv', index=False, encoding='utf-8')

Pandas只写入CSV文件中的最后一行

1 个答案: