Question

我有一个循环遍历多个页面的脚本，它正常工作，直到它到达第19页。我收到错误消息：

if container.find(text="License: ").nextSibling.img:
AttributeError: 'NoneType' object has no attribute 'nextSibling'

该元素存在于第19页。我还包含了一个else语句，当时它没有。我尝试过requests而不是urlopen，看看是否有所改变，甚至将解析器从html.parser更改为html5lib到lxml，但没有运气。我认为它可能是一个解析器问题，但我不确定解决方案可能是什么。这是我的剧本：

from urllib.request import urlopen
from bs4 import BeautifulSoup as soup
import json

base_url = "https://www.doabooks.org/"

data = []
n = 85
for i in range(1, n+1):
    if (i == 1):
    # handle first page
       response = urlopen(base_url)
    response = urlopen(base_url + "doab?func=browse&page=" + str(i) + "&queryField=A&uiLanguage=en")
    page_html = response.read()
    response.close()

    #html parsing
    page_soup = soup(page_html, "html5lib")

    #grabs info for each book
    containers = page_soup.findAll("div",{"class":"data"})

    for container in containers:
       item = {}
       item['type'] = "Open Access Book"
       item['title'] = container.span.text.strip()
       item['author'] =container.a.text
       item['link'] = "https://www.doabooks.org" + container.find('a', {'itemprop' : 'url'})['href']
       item['source'] = "Directory of Open Access Books"
       if container.div.find('a', {'itemprop' : 'about'}):
          item['subject'] = container.div.find('a', {'itemprop' : 'about'}).text.lstrip()
       else: 
          item['subject'] = ''
       item['base_url'] = "https://www.doabooks.org/"
       if container.find(text="License: ").nextSibling.img:
           item['license'] = container.find(text="License: ").nextSibling['href']
       else:
           item['license'] = container.find(text="License: ").nextSibling.text

       item['license_url'] = container.find(text="License: ").nextSibling['href']
       data.append(item) # add the item to the list

   with open("./json/doab-a.json", "w") as writeJSON:
      json.dump(data, writeJSON, ensure_ascii=False)

Answer 1

标题为＃34;转换后的书＃34;在https://www.doabooks.org/doab?func=browse&page=19&queryField=A&uiLanguage=en上没有＆＃34;许可证＆＃34;给定，所以在你的代码中

container.find(text="License: ")

是无，所以你无法获得

.nextSibling.img

来自NoneType对象的

因此抛出异常。尝试这样的事情：

if not container.find(string="License: "):
    item['license_url'] = item['license'] = "Not Specified"
else:
    if container.find(string="License: ").nextSibling.img:
        item['license'] = container.find(string="License: ").nextSibling['href']
    else:
        item['license'] = container.find(string="License: ").nextSibling.text
    item['license_url'] = container.find(string="License: ").nextSibling['href']

BeautifulSoup的解析器问题

1 个答案: