Question

我正在学习Python中的美丽汤和词典。我正在遵循斯坦福大学美丽汤的简短教程，可以在这里找到：http://web.stanford.edu/~zlotnick/TextAsData/Web_Scraping_with_Beautiful_Soup.html

由于访问webside是禁止的，我已将教程中显示的文本存储到字符串中，然后将字符串汤转换为汤对象。打印输出如下：

    print(soup_string)

    <html><body><div class="ec_statements"><div id="legalert_title"><a    
    href="/Legislation-and-Politics/Legislative-Alerts/Letter-to-Senators-
    Urging-Them-to-Support-Cloture-and-Final-Passage-of-the-Paycheck-
    Fairness-Act-S.2199">'Letter to Senators Urging Them to Support Cloture     
    and Final Passage of the Paycheck Fairness Act (S.2199)
    </a>
    </div>
    <div id="legalert_date">
    September 10, 2014
    </div>
    </div>
    <div class="ec_statements">
    <div id="legalert_title">
    <a href="/Legislation-and-Politics/Legislative-Alerts/Letter-to-  
    Representatives-Urging-Them-to-Vote-on-the-Highway-Trust-Fund-Bill">
    Letter to Representatives Urging Them to Vote on the Highway Trust Fund Bill
    </a>
    </div>
    <div id="legalert_date">
            July 30, 2014
           </div>
    </div>
    <div class="ec_statements">
    <div id="legalert_title">
    <a href="/Legislation-and-Politics/Legislative-Alerts/Letter-to-Representatives-Urging-Them-to-Vote-No-on-the-Legislation-Providing-Supplemental-Appropriations-for-the-Fiscal-Year-Ending-Sept.-30-2014">
             Letter to Representatives Urging Them to Vote No on the Legislation Providing Supplemental Appropriations for the Fiscal Year Ending Sept. 30, 2014
            </a>
    </div>
    <div id="legalert_date">
            July 30, 2014
           </div>
    </div>
 </body></html>

在某些时候，导师捕获汤对象中具有Tag“div”，class _ =“ec_statements”的所有元素。

   letters = soup_string.find_all("div", class_="ec_statements")

然后导师说：

“我们将查看我们的字母集合中的所有项目，并为每个项目提取名称并将其作为我们字典中的密钥。该值将是另一个字典，但我们还没有找到其他项目的内容，所以我们只需创建一个空的dict对象。“

此时我采用了不同的方法，我决定先将数据存储在列表中，然后存储在数据帧中。代码如下：

lobbying_1 = []
lobbying_2 = []
lobbying_3 = []
for element in letters:
    lobbying_1.append(element.a.get_text())
    lobbying_2.append(element.a.attrs.get('href'))
    lobbying_3.append(element.find(id="legalert_date").get_text())
df =pd.DataFrame([])
df = pd.DataFrame(lobbying_1, columns = ['Name'] )
df['href'] = lobbying_2
df['Date'] = lobbying_3

输出如下：

print(df)

                                                Name  \
0  \n        'Letter to Senators Urging Them to S...   
1  \n         Letter to Representatives Urging Th...   
2  \n         Letter to Representatives Urging Th...   

                                                href  \
0  /Legislation-and-Politics/Legislative-Alerts/L...   
1  /Legislation-and-Politics/Legislative-Alerts/L...   
2  /Legislation-and-Politics/Legislative-Alerts/L...   

                                    Date  
0  \n        September 10, 2014\n         
1       \n        July 30, 2014\n         
2       \n        July 30, 2014\n

我的问题是：有没有办法获得更清晰的数据，即没有\ n和空格的字符串，只有通过Beautiful Soup的真实值？或者我必须使用Regex发布数据处理？

您的建议将不胜感激。

Answer 1

要删除文本中的换行符，请在调用strip=True时传递get_text()：

for element in letters:
    lobbying_1.append(element.a.get_text(strip=True))
    lobbying_2.append(element.a.attrs.get('href'))
    lobbying_3.append(element.find(id="legalert_date").get_text(strip=True))

当然，这假设您仍然希望数据采用DataFrame的形式。

获取干净的数据：美丽的汤就够了，或者我也必须使用正则表达式？

1 个答案: