Question

以下程序为我提供的输出包括带有和不带正斜杠的URL（例如ask.census.gov和ask.census.gov/）。我需要消除其中一个。提前感谢您的帮助！

from bs4 import BeautifulSoup as mySoup
from urllib.parse import urljoin as myJoin
from urllib.request import urlopen as myRequest

my_url = "https://www.census.gov/programs-surveys/popest.html"

# call on packages
html_page = myRequest(my_url)
raw_html = html_page.read()
html_page.close()
page_soup = mySoup(raw_html, "html.parser")

f = open("censusTest.csv", "w")

hyperlinks = page_soup.findAll('a')

set_urls = set()

for checked in hyperlinks:
    found_link = checked.get("href")
    result_set = myJoin(my_url, found_link)
    if result_set and result_set not in set_urls:
        set_urls.add(result_set)
        f.write(str(result_set) + "\n")

f.close()

Answer 1

您可以随时right-strip the slash - 如果存在则会被删除，否则不会执行任何操作：

result_set = myJoin(my_url, found_link).rstrip("/")

Answer 2

my_url = "https://www.census.gov/programs-surveys/popest.html/"
if my_url[-1:] == '/':
    my_url = my_url[:-1]

这段代码会检查字符串中的最后一个字符是否为＆＃39; /＆＃39;，如果是，则会将其删除。

python字符串操作的好例子： http://www.pythonforbeginners.com/basics/string-manipulation-in-python

删除Python中的重复URL，包括包含正斜杠的URL

2 个答案: