网页抓取 html 页面时如何处理无类型对象

Question

我想从 rottentomatoes 中抓取一个页面

我试图找出不同电影的所有导演的名字。到目前为止，我的代码运行良好。但是，网页上有一部名为 WORLD ON A WIRE 的电影。这部电影缺少导演的名字。现在，当我运行代码时，它给了我像 NoneType object is not iterable 这样的错误。现在我如何在抓取网页时处理空字段。

我的代码：

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
url= 'https://editorial.rottentomatoes.com/guide/best-sci-fi-movies-of-all-time/'
r = requests.get(url, headers=headers)#, proxies=proxies)
content = r.content
soup = BeautifulSoup(content)
director = []
for d in soup.find_all('div', attrs={'class': 'info director'}):
    for a in d.find('a'):
        director.append(a)
        print(a)

Answer 1

d.find('a') 在你的代码中没有返回一个可迭代对象，这会导致 python 错误。您应该使用 find_all('a') 而不是 find('a')。

您的代码应如下所示：

headers = {"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", "Accept-Encoding":"gzip, deflate", "Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "DNT":"1","Connection":"close", "Upgrade-Insecure-Requests":"1"}
url= 'https://editorial.rottentomatoes.com/guide/best-sci-fi-movies-of-all-time/'
r = requests.get(url, headers=headers)#, proxies=proxies)
content = r.content
soup = BeautifulSoup(content)
director = []
for d in soup.find_all('div', attrs={'class': 'info director'}):
    for a in d.find_all('a'):
        director.append(a.string)
        print(a.string)