我正在尝试获取特定月份(例如网站上的 3 月)中的所有链接、标题和日期,我正在使用 BeautifulSoup
来执行此操作:
from bs4 import BeautifulSoup
import requests
html_link='https://www.pds.com.ph/index.html%3Fpage_id=3261.html'
html = requests.get(html_link).text
soup = BeautifulSoup(html, 'html.parser')
for link in soup.find_all('td'):
#Text contains 'March'
#Get the link title & link &date
我是 BeautifulSoup
的新手,在 Selenium
中我使用了 xpath:"//td[contains(text(),'Mar')"
,我如何使用 BeautifulSoup
执行此操作?
答案 0 :(得分:4)
如果“日期”包含文本“三月”,则获取所有链接和标题:
查找“日期” - 找到所有包含文本“march”的 <td>
元素。
使用包含所需标题和链接的 .find_previous()
方法查找前一个 <a>
标记。
import requests
from bs4 import BeautifulSoup
url = "https://www.pds.com.ph/index.html%3Fpage_id=3261.html"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
fmt_string = "{:<20} {:<130} {}"
print(fmt_string.format("Date", "Title", "Link"))
print('-' * 200)
for tag in soup.select("td:contains('March')"):
a_tag = tag.find_previous("a")
print(
fmt_string.format(
tag.text, a_tag.text, "https://www.pds.com.ph/" + a_tag["href"],
)
)
输出(截断):
Date Title Link
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
March 31, 2021 RCBC Lists PHP 17.87257 Billion ASEAN Sustainability Bonds on PDEx https://www.pds.com.ph/index.html%3Fp=87239.html
March 16, 2021 Aboitiz Power Corporation Raises 8 Billion Fixed Rate Bonds on PDEx https://www.pds.com.ph/index.html%3Fp=86743.html
March 1, 2021 Century Properties Group, Inc Returns to PDEx with PHP 3 Billion Fixed Rate Bonds https://www.pds.com.ph/index.html%3Fp=86366.html
March 27, 2020 BPI Lists Over PhP 33 Billion of Fixed Rate Bonds on PDEx https://www.pds.com.ph/index.html%3Fp=74188.html
March 25, 2020 SM Prime Raises PHP 15 Billion Fixed Rate Bonds on PDEx https://www.pds.com.ph/index.html%3Fp=74082.html
...
答案 1 :(得分:3)
这是您可以尝试的解决方案,
import re
import requests
from bs4 import BeautifulSoup
html_link = 'https://www.pds.com.ph/index.html%3Fpage_id=3261.html'
html = requests.get(html_link).text
soup = BeautifulSoup(html, 'html.parser')
search = re.compile("March")
for td in soup.find_all('td', text=search):
link = td.parent.select_one("td > a")
if link:
print(f"Title : {link.text}")
print(f"Link : {link['href']}")
print(f"Date : {td.text}")
print("-" * 30)
Title : RCBC Lists PHP 17.87257 Billion ASEAN Sustainability Bonds on PDEx
Link : index.html%3Fp=87239.html
Date : March 31, 2021
------------------------------
Title : Aboitiz Power Corporation Raises 8 Billion Fixed Rate Bonds on PDEx
Link : index.html%3Fp=86743.html
Date : March 16, 2021
------------------------------
....