这些是规则
<p>
,<ol>
或<ul>
<em>
,<strong>
或<span style="text-decoration:underline">
<strong>
将是JSON中的此项{"bold":True}
,<em>
将{"italics":True}
和<span style="text-decoration:underline">
将{"decoration":"underline"}
1}} {"text": "this is the text"}
醇>
假设我有下面的HTML:使用这个:
soup = Soup("THIS IS THE WHOLE HTML", "html.parser")
allTags = [tag for tag in soup.find_all(recursive=False)]
产生这个数组:
[
<p>The name is not mine it is for the people<span style="text-decoration: underline;"><em><strong>stephen</strong></em></span><em><strong> how can</strong>name </em><strong>good</strong> <em>his name <span style="text-decoration: underline;">moneuet</span>please </em><span style="text-decoration: underline;"><strong>forever</strong></span><em>tomorrow<strong>USA</strong></em></p>,
<p>2</p>,
<p><strong>moment</strong><em>Africa</em> <em>China</em> <span style="text-decoration: underline;">home</span> <em>thomas</em> <strong>nothing</strong></p>,
<ol><li>first item</li><li><em><span style="text-decoration: underline;"><strong>second item</strong></span></em></li></ol>
]
通过应用上述规则,结果如下:
第一个数组元素将被处理成这个JSON:
{
"text": [
"The name is not mine it is for the people",
{"text": "stephen", "decoration": "underline", "bold": True, "italics": True},
{"text": "how can", "bold": True, "italics": True},
{"text": "name", "italics": True},
{"text": "good", "bold": True},
{"text": "his name", "italics": True},
{"text": "moneuet", "decoration": "underline"},
{"text": "please ", "italics": True},
{"text": "forever", "decoration": "underline", "bold":True},
{"text": "tomorrow", "italics": True},
{"text": "USA", "bold": True, "italics": True}
]
}
第二个数组元素将被处理成这个JSON:
{"text": ["2"] }
第三个数组元素将被处理成这个JSON:
{
"text": [
{"text": "moment", "bold": True},
{"text": "Africa", "italics": True},
{"text": "China", "italics": True},
{"text": "home", "decoration": "underline"},
{"text": "thomas", "italics": True},
{"text": "nothing", "bold": True}
]
}
第四个Array元素将被处理成这个JSON:
{
"ol": [
"first item",
{"text": "second item", "decoration": "underline", "italics": True, "bold": True}
]
}
这是我的尝试,我可以向下钻取。但是如何处理arrayOfTextAndStyles数组是个问题
soup = Soup("THIS IS THE WHOLE HTML", "html.parser")
allTags = [tag for tag in soup.find_all(recursive=False)]
for foundTag in allTags:
foundTagStyles = [tag for tag in foundTag.find_all(recursive=True)]
if len(foundTagStyles ) > 0:
if str(foundTag.name) == "p":
arrayOfTextAndStyles = [{"tag": tag.name, "text":
foundTag.find_all(text=True, recursive=False) }] +
[{"tag":tag.name, "text": foundTag.find_all(text=True,
recursive=False) } for tag in foundTag.find_all()]
elif str(foundTag.name) == "ol":
elif str(foundTag .name) == "ul":
答案 0 :(得分:1)
我使用函数来解析每个元素,而不是使用一个巨大的循环。选择p
和ol
代码,并在解析中引发异常,以标记与您的特定规则不匹配的任何内容:
from bs4 import NavigableString
def parse(elem):
if elem.name == 'ol':
result = []
for li in elem.find_all('li'):
if len(li) > 1:
result.append([parse_text(sub) for sub in li])
else:
result.append(parse_text(next(iter(li))))
return {'ol': result}
return {'text': [parse_text(sub) for sub in elem]}
def parse_text(elem):
if isinstance(elem, NavigableString):
return {'text': elem}
result = {}
if elem.name == 'em':
result['italics'] = True
elif elem.name == 'strong':
result['bold'] = True
elif elem.name == 'span':
try:
# rudimentary parse into a dictionary
styles = dict(
s.replace(' ', '').split(':')
for s in elem.get('style', '').split(';')
if s.strip()
)
except ValueError:
raise ValueError('Invalid structure')
if 'underline' not in styles.get('text-decoration', ''):
raise ValueError('Invalid structure')
result['decoration'] = 'underline'
else:
raise ValueError('Invalid structure')
if len(elem) > 1:
result['text'] = [parse_text(sub) for sub in elem]
else:
result.update(parse_text(next(iter(elem))))
return result
然后您解析文档:
for candidate in soup.select('ol,p'):
try:
result = parse(candidate)
except ValueError:
# invalid structure, ignore
continue
print(result)
使用pprint
,结果如下:
{'text': [{'text': 'The name is not mine it is for the people'},
{'bold': True,
'decoration': 'underline',
'italics': True,
'text': 'stephen'},
{'italics': True,
'text': [{'bold': True, 'text': ' how can'}, {'text': 'name '}]},
{'bold': True, 'text': 'good'},
{'text': ' '},
{'italics': True,
'text': [{'text': 'his name '},
{'decoration': 'underline', 'text': 'moneuet'},
{'text': 'please '}]},
{'bold': True, 'decoration': 'underline', 'text': 'forever'},
{'italics': True,
'text': [{'text': 'tomorrow'}, {'bold': True, 'text': 'USA'}]}]}
{'text': [{'text': '2'}]}
{'text': [{'bold': True, 'text': 'moment'},
{'italics': True, 'text': 'Africa'},
{'text': ' '},
{'italics': True, 'text': 'China'},
{'text': ' '},
{'decoration': 'underline', 'text': 'home'},
{'text': ' '},
{'italics': True, 'text': 'thomas'},
{'text': ' '},
{'bold': True, 'text': 'nothing'}]}
{'ol': [{'text': 'first item'},
{'bold': True,
'decoration': 'underline',
'italics': True,
'text': 'second item'}]}
请注意,文本节点现在是嵌套;这使您可以使用正确的空格和嵌套的文本装饰来一致地重新创建相同的结构。
结构也相当一致; 'text'
键将指向一个字符串,或一个字典列表。这样的列表永远不会混合类型。你可以改善这一点;让'text'
仅指向一个字符串,并使用不同的键来表示嵌套数据,例如contains
或nested
或类似,然后只使用一个或另一个。所有这些都需要更改'text'
案例和len(elem) > 1
函数中的parse()
个键。