Question

我想从一个自然语言处理项目的医学文档的网页上获取一些文本，并且在使用BeautifulSoup提取必要的信息时遇到问题。我正在查看的网站可以在以下地址找到：https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D

我想做的是从此页面上抓取整个文本主体，并使用光标进行操作，只需应用复制/粘贴即可为我提供我感兴趣的适当文本：

Sample Type / Medical Specialty: Gastroenterology
Sample Name: Abdominal Abscess I&D
Description: Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.
(Medical Transcription Sample Report)
PREOPERATIVE DIAGNOSIS: Abdominal wall abscess.

... (body text) ...

The finished wound size was 9.0 x 5.3 x 5.2 cm in size. Patient tolerated the procedure well. Dressing was applied, and he was taken to recovery room in stable condition.

但是，我想使用BeautifulSoup来实现这一点，因为我想执行一个循环以从同一网站获取多个医疗文档。

import requests  
r = requests.get('https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D')

from bs4 import BeautifulSoup  
soup = BeautifulSoup(r.text, 'html.parser')  
results = soup.find_all('div', attrs={'id':'sampletext'})

# Here I am able to specify the <h1> tag to get 'Sample Type / Medical Specialty' as well as 'Sample Name' text fields

record.find('h1').text.replace('\n', ' ')

但是，我无法为其余文本（即说明，术前诊断，术后诊断，手术等）复制此文本，因为没有唯一的标签来标识这些文本字段

如果任何人都熟悉使用BeautifulSoup进行网络抓取的概念，我将不胜感激！同样，我的目标是从网页中获取全文，最终我希望将其添加到Pandas Dataframe中。谢谢！

Answer 1

好吧，花了我一段时间，但是除非您手动遍历所有元素，否则没有一种简单的方法来提取可用的文本：

import requests
import re
from bs4 import BeautifulSoup, Tag, NavigableString, Comment

url = 'https://www.mtsamples.com/site/pages/sample.asp?Type=24-Gastroenterology&Sample=2332-Abdominal%20Abscess%20I&D'
res = requests.get(url)
res.raise_for_status()
html = res.text
soup = BeautifulSoup(html, 'html.parser')

到目前为止没有什么特别的。

title_el = soup.find('h1')
page_title = title_el.text.strip()
first_hr = title_el.find_next_sibling('hr')

description_title = title_el.find_next_sibling('b', text=re.compile('description', flags=re.I))
description_text_parts = []
for s in description_title.next_siblings:
    if s is first_hr:
        break
    if isinstance(s, Tag):
        description_text_parts.append(s.text.strip())
    elif isinstance(s, NavigableString):
        description_text_parts.append(str(s).strip())
description_text = '\n'.join(p for p in description_text_parts if p.strip())

我们从page_title那里获得<h1>

'Sample Type / Medical Specialty:  Gastroenterology\nSample Name: Abdominal Abscess I&D'

和description，我们会在看到文本Description:之后在元素之间移动。

'Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.\n(Medical Transcription Sample Report)'

现在，所有标题均置于水平线以下：

# titles are all bold and uppercase
titles = [b for b in first_hr.find_next_siblings('b') if b.text.strip().isupper()]

我们在标题之间找到文本，并将其分配给我们之前看到的标题

docs = []
for t in titles:
    text_parts = []
    for s in t.next_siblings:
        # go until next title
        if s in titles:
            break
        if isinstance(s, Comment):
            continue
        if isinstance(s, Tag):
            if s.name == 'div':
                break
            text_parts.append(s.text.strip())
        elif isinstance(s, NavigableString):
            text_parts.append(str(s).strip())
    text = '\n'.join(p for p in text_parts if p.strip())
    docs.append({
        'title': t.text.strip(),
        'text': text
    })

打印文档给出：

[
{'title': 'PREOPERATIVE DIAGNOSIS:', 'text': 'Abdominal wall abscess.'}, 
{'title': 'POSTOPERATIVE DIAGNOSIS:', 'text': 'Abdominal wall abscess.'}, 
{'title': 'PROCEDURE:', 'text': 'Incision and drainage (I&D) of abdominal abscess, excisional debridement of nonviable and viable skin, subcutaneous tissue and muscle, then removal of foreign body.'}, 
{'title': 'ANESTHESIA:', 'text': 'LMA.'}, 
{'title': 'INDICATIONS:', 'text': 'Patient is a pleasant 60-year-old gentleman, who initially had a sigmoid colectomy for diverticular abscess, subsequently had a dehiscence with evisceration.  Came in approximately 36 hours ago with pain across his lower abdomen.  CT scan demonstrated presence of an abscess beneath the incision.  I recommended to the patient he undergo the above-named procedure.  Procedure, purpose, risks, expected benefits, potential complications, alternatives forms of therapy were discussed with him, and he was agreeable to surgery.'}, 
{'title': 'FINDINGS:', 'text': 'The patient was found to have an abscess that went down to the level of the fascia.  The anterior layer of the fascia was fibrinous and some portions necrotic.  This was excisionally debrided using the Bovie cautery, and there were multiple pieces of suture within the wound and these were removed as well.'},
{'title': 'TECHNIQUE:', 'text': 'Patient was identified, then taken into the operating room, where after induction of appropriate anesthesia, his abdomen was prepped with Betadine solution and draped in a sterile fashion.  The wound opening where it was draining was explored using a curette.  The extent of the wound marked with a marking pen and using the Bovie cautery, the abscess was opened and drained.  I then noted that there was a significant amount of undermining.  These margins were marked with a marking pen, excised with Bovie cautery; the curette was used to remove the necrotic fascia.  The wound was irrigated; cultures sent prior to irrigation and after achievement of excellent hemostasis, the wound was packed with antibiotic-soaked gauze.  A dressing was applied.  The finished wound size was 9.0 x 5.3 x 5.2 cm in size.  Patient tolerated the procedure well.  Dressing was applied, and he was taken to recovery room in stable condition.'}
]

如何使用BeautifulSoup从网页上获取整个正文？

1 个答案: