Question

我正在尝试从此webpage中提取数据，并且由于页面的HTML格式不一致而导致出现问题。我有一个OGAP ID列表，我想为每个迭代的OGAP ID提取基因名称和任何文献信息（PMID＃）。感谢此处的其他问题以及BeautifulSoup文档，我已经能够为每个ID一致地获取基因名称，但我在文献部分遇到了问题。这是一些突出不一致的搜索字词。

有效的HTML示例

搜索字词：OG00131

＆＃13;

<tr>
  <td colspan="4" bgcolor="#FBFFCC" class="STYLE28">Literature describing O-GlcNAcylation:
    <br>&nbsp;&nbsp;PMID: 
    <a href="http://www.ncbi.nlm.nih.gov/pubmed/20068230">20068230</a>
    [CAD, ETD MS/MS]; <br>
    <br>
  </td>
</tr>

＆＃13;

不起作用的HTML示例

搜索字词：OG00020

＆＃13;

<td align="top" bgcolor="#FBFFCC">
  <div class="STYLE28">Literature describing O-GlcNAcylation: </div>
  <div class="STYLE28">
    <div class="STYLE28">PMID:
      <a href="http://www.ncbi.nlm.nih.gov/pubmed/16408927?dopt=Citation">16408927</a>
      [Azide-tag, nano-HPLC/tandem MS] 
    </div>
    <br>
    Site has not yet been determined. Use 
    <a href="parser2.cgi?ACLY_HUMAN" target="_blank">OGlcNAcScan</a>
    to predict the O-GlcNAc site. </div>
</td>

＆＃13;

这是我到目前为止的代码

import urllib2
from bs4 import BeautifulSoup

#define list of genes

#initialize variables
gene_list = []
literature = []
# Test list
gene_listID = ["OG00894", "OG00980", "OG00769", "OG00834","OG00852", "OG00131","OG00020"]


for i in range(len(gene_listID)):
    print gene_listID[i]
    # Specifies URL, uses the "%" to sub in different ogapIDs based on a list provided
    dbOGAP = "https://wangj27.u.hpc.mssm.edu/cgi-bin/DB_tb.cgi?textfield=%s&select=Any" % gene_listID[i]
    # Opens the URL as a page
    page = urllib2.urlopen(dbOGAP)
    # Reads the page and parses it through "lxml" format
    soup = BeautifulSoup(page, "lxml")

    gene_name = soup.find("td", text="Gene Name").find_next_sibling("td").text
    print gene_name[1:]
    gene_list.append(gene_name[1:])

    # PubMed IDs are located near the <td> tag with the term "Data and Source"
    pmid = soup.find("span", text="Data and Source")

    # Based on inspection of the website, need to move up to the parent <td> tag
    pmid_p = pmid.parent

    # Then we move to the next <td> tag, denoted as sibling (since they share parent <tr> (Table row) tag)
    pmid_s = pmid_p.next_sibling
    #for child in pmid_s.descendants:
     #   print child
    # Now we search down the tree to find the next table data (<td>) tag
    pmid_c = pmid_s.find("td")
    temp_lit = []
    # Next we print the text of the data
    #print pmid_c.text
    if "No literature is available" in pmid_c.text:
        temp_lit.append("No literature is available")
        print "Not available"
    else:
    # and then print out a list of urls for each pubmed ID we have
        print "The following is available"
        for link in pmid_c.find_all('a'):
            # the <a> tag includes more than just the link address.
            # for each <a> tag found, print the address (href attribute) and extra bits
            # link.string provides the string that appears to be hyperlinked.
            # In this case, it is the pubmedID
            print link.string
            temp_lit.append("PMID: " + link.string + "  URL: " + link.get('href'))
    literature.append(temp_lit)
    print "\n"

所以看起来元素就是抛出循环的代码。有没有办法用文本＆＃34; PMID＆＃34;搜索任何元素？并返回它后面的文本（如果有PMID号，则返回url）？如果没有，我是否只想检查每个孩子，寻找我感兴趣的文本？

我使用的是Python 2.7.10

Answer 1

import requests
from bs4 import BeautifulSoup
import re
gene_listID = ["OG00894", "OG00980", "OG00769", "OG00834","OG00852", "OG00131","OG00020"]
urls = ('https://wangj27.u.hpc.mssm.edu/cgi-bin/DB_tb.cgi?textfield={}&select=Any'.format(i) for i in gene_listID)

for url in urls: 
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'lxml')
    regex = re.compile(r'http://www.ncbi.nlm.nih.gov/pubmed/\d+')

    a_tag = soup.find('a', href=regex)
    has_pmid = 'PMID' in a_tag.previous_element

    if has_pmid :
        print(a_tag.text, a_tag.next_sibling, a_tag.get("href"))
    else:
        print("Not available")

出：

18984734  [GalNAz-Biotin tagging, CAD MS/MS];  http://www.ncbi.nlm.nih.gov/pubmed/18984734
20068230  [CAD, ETD MS/MS];  http://www.ncbi.nlm.nih.gov/pubmed/20068230
20068230  [CAD, ETD MS/MS];  http://www.ncbi.nlm.nih.gov/pubmed/20068230
Not available
16408927  [Azide-tag, nano-HPLC/tandem MS];   http://www.ncbi.nlm.nih.gov/pubmed/16408927
Not available
16408927 [Azide-tag, nano-HPLC/tandem MS]  http://www.ncbi.nlm.nih.gov/pubmed/16408927?dopt=Citation

找到第一个与目标网址匹配的标记，以数字结尾，而不是检查其前一个元素中的“PMID”。这个网络是如此不和谐，我尝试了很多次，希望这会有所帮助

使用BeautifulSoup4和Python从不一致的HTML页面中提取数据

1 个答案: