由于网页中的广告，使用Beautiful汤来抓取网站会产生大量空白

Question

以下是我尝试抓取的链接： Livemint news

以下是尝试实现此功能的功能：

    t = []
try:
    temp = []
    data = bs.find_all(class_=['contentSec'])
    # logging.info(data)
    for i in data:
        temp = temp + (i.find_all('p'))
    for i in temp:
        t.append(i.get_text())
except Exception as e:
    print(e)
return t

发生的事情是，如果我包括的话，在find all参数中，text = True，它将忽略带有链接（带有href标记）的paras。 否则，它在内容字段中给了我巨大的空白，可能是因为网站中的广告也位于para标签中。我已附上示例输出。

我想念什么？

Answer 1

您要查找的数据（即文章的内容）可直接在页面div下的类为mainArea的页面中获得。您所要做的就是获取该div的文本并进行清理。对于您所需的数据，我认为根本不需要找到script标记并使用json模块。但是，如果您需要datePublished之类的数据，则@ chitown88的答案会更全面。

from bs4 import BeautifulSoup
import requests
url='https://www.livemint.com/Companies/Ot1UTmQ8EMe0DTWSiJCgfJ/Google-teams-with-HDFC-Bank-ICICI-others-for-instant-loans.html'
r=requests.get(url)
soup=BeautifulSoup(r.text,'html.parser')
data=soup.find('div',class_='mainArea')
#let's just clen up the data
cleaned_data=data.text.split('\n\n')[0].strip()
print(cleaned_data)

输出

New Delhi: Google India on Tuesday said it has rebranded its Indian payments app Tez as Google Pay and is partnering four banks to provide instant loans for the app’s users. In the coming weeks, Google Pay users will be able to access customised loans from HDFC Bank Ltd, ICICI Bank Ltd, Federal Bank and Kotak Mahindra Bank Ltd with minimal paperwork, said Caesar Sengupta, vice-president of Google’s Next Billion Users Initiative and Payments, at the Google for India event in New Delhi. Once users holding accounts with these banks accept the bank’s terms, the money will be transferred to their accounts.“We have learnt that when we build for India, we build for the world, and we believe that many of the innovations and features we have pioneered with Tez will work globally," Caesar Sengupta said.Google Tez, which was launched in September, will also expand services for merchants and retailers. About 15,000 retail stores in India will have Google Pay enabled by Diwali 2018, Caesar Sengupta said.Google claims that over 1.2 million small businesses in India are already using Google Pay. In a bid to help their business grow further, Google is building a dedicated merchant experience where they will be discovered through Google Search and Maps, and communicate with their customers through messages and offers.“We are testing these features with merchants in Bangalore and Delhi, and on-boarding more neighbourhoods in the following months," said Sengupta.Google Pay has rivals in Paytm and Facebook Inc.’s WhatsApp targeting the Indian payments market. On Tuesday, Mint reported that Warren Buffett’s Berkshire Hathaway Inc. has sealed a deal with Paytm, marking the legendary investor’s first investment in the country. A string of other big-name players are also expanding in India’s digital payments market including its banks, India Post Payments Bank, and Mukesh Ambani’s Reliance Jio.“The real competition is actually user habits and cash," said Sengupta. “So, all of us (referring to the other players) are all in many ways brothers-in-arms who are trying to move people’s habits away from cash to digital so that we can move India to a digital economy. At Google, we focus on the users so we don’t think so much of the competition."Since its launch, over 55 million people have downloaded Google Tez and more than 22 million people and businesses actively use the app for digital transactions every month, according to company data and figures quoted by Sengupta. Collectively, they have made more than 750 million transactions, with an annual run rate of over $30 billion.The search giant also announced other initiatives including expanding its Google Station internet access programme to 12,000 villages and cities across Andhra Pradesh, potentially reaching 10 million people; the launch of Project Navlekha, where Google will work with Indian publishers to bring more relevant content online; and a new feature in Google Go app that can pull up any webpage and let users listen to it with each word lighting up as it is read.

Answer 2

import requests
import bs4
import pandas as pd
import json

list_of_urls = ['https://www.livemint.com/Companies/Ot1UTmQ8EMe0DTWSiJCgfJ/Google-teams-with-HDFC-Bank-ICICI-others-for-instant-loans.html']

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'}

results_df = pd.DataFrame()


for url in list_of_urls:
    response = response = requests.get(url, headers=headers)
    soup = bs4.BeautifulSoup(response.text, 'html.parser')

    scripts = soup.find_all('script',{'type':'application/ld+json'})
    for script in scripts:
        if '"headline"' in script.text:
            jsonStr = script.text.strip()
            jsonObj = json.loads(jsonStr)

            date_pub = jsonObj['datePublished']
            date_mod = jsonObj['dateModified']
            data = jsonObj['articleBody']
            url = jsonObj['url']

            temp_df = pd.DataFrame([[date_pub, data, url]], columns=['date_published','data','url'])

            results_df = results_df.append(temp_df)

results_df = results_df.reset_index(drop=True)
results_df.to_csv('path/to/file.csv', index=False)