更新

Question

尝试将this页面上的条目抓取为制表符分隔的格式（主要是提取序列和UniProt登录号）。

当我跑步时：

url = 'www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=1000&listname='    
table = pd.read_html(url)
print(table)

我得到：

Traceback (most recent call last):
  File "scrape_signalpeptides.py", line 7, in <module>
    table = pd.read_html(url)
  File "/Users/ION/anaconda3/lib/python3.7/site-packages/pandas/io/html.py", line 1094, in read_html
    displayed_only=displayed_only)
  File "/Users/ION/anaconda3/lib/python3.7/site-packages/pandas/io/html.py", line 916, in _parse
    raise_with_traceback(retained)
  File "/Users/ION/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 420, in raise_with_traceback
    raise exc.with_traceback(traceback)
ValueError: No tables found

所以我尝试了漂亮的汤法：

import requests
import pandas as pd
import json
from pandas.io.json import json_normalize
from bs4 import BeautifulSoup

url = 'http://www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=1000&listname='
res = requests.get(url)
soup = BeautifulSoup(res.content, "lxml")
print(soup)

我可以看到那里有数据。有谁知道为什么我不能使用pandas.read_html解析此页面？编辑1：根据以下建议，我运行了此

：

from bs4 import BeautifulSoup
import requests
s = requests.session()
s.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36'
res = s.get('https://www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=2&listname=')
print(res)

....我将URL更改为所有www，http和https；对于所有错误，我都会得到与连接错误有关的错误，例如

urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x1114f0898>: Failed to establish a new connection: [Errno 61] Connection refused

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.signalpeptide.de', port=443): Max retries exceeded with url: /index.php?sess=&m=listspdb_bacteria&s=details&id=2&listname= (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x1114f0898>: Failed to establish a new connection: [Errno 61] Connection refused'

ConnectionRefusedError: [Errno 61] Connection refused

Answer 1

两个脚本中的url变量不同。

并排比较：

url = 'www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=1000&listname=' # pandas
url = 'http://www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=1000&listname=' # BeautifulSoup

我怀疑http://位对于大熊猫将其识别为URL而不是HTML本身很重要。毕竟，pandas.read_html动态地将参数解释为described in the documentation

URL，类似文件的对象或包含HTML的原始字符串。请注意，lxml仅接受http，ftp和文件url协议。如果您有一个以“ https”开头的URL，则可以尝试删除“ s”。

特别是部分，如果您有一个以“ https”开头的网址，则可以尝试删除“ s” ，这使我相信http://对于了解它很重要是一个链接，而不是“类似文件的对象”或原始HTML。

如果错误超过最大重试次数，则可能需要实现带有标头的requests.session。我为此所做的先前代码如下：

import requests s = requests.session() s.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36' res = s.get('your_url')

此时，您应该能够以与普通res对象相同的方式解释requests.get()对象（可以调用.text之类的方法）。我不太确定s.headers的工作方式，只是从另一个SO帖子复制并修复了我的脚本！

更新

最后一个代码块中的部分错误消息是

ssl.CertificateError：主机名“ www.signalpeptide.de”与“ www.kg13.art”，“ www.thpr.net”都不匹配

这意味着他们的SSL证书无效，并且https可能无法工作，因为无法验证主机。我将其调整为http并显示结果HTML：

from bs4 import BeautifulSoup import requests s = requests.session() s.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36' res = s.get('http://www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=2&listname=') print(res.text)

结果：

C:\Users\rparkhurst\PycharmProjects\Workspace\venv\Scripts\python.exe C:/Users/rparkhurst/PycharmProjects/Workspace/new_workspace.py <!doctype html> <html class="no-js" lang="en"> <head> <meta charset="utf-8"/> <meta name="viewport" content="width=device-width, initial-scale=1.0"/> <title>Signal Peptide Database</title> <link rel="stylesheet" href="css/foundation.css"> <link href='http://cdnjs.cloudflare.com/ajax/libs/foundicons/3.0.0/foundation-icons.css' rel='stylesheet' type='text/css'> <link href="css/custom.css" rel="stylesheet" type="text/css"> </head> <body> <div class="top-bar"> <div class="row"> <div class="top-bar-left"> <div class="top-bar-title"> <span data-responsive-toggle="responsive-menu" data-hide-for="medium"> <span class="menu-icon dark" data-toggle></span> </span> <a href="./"><img src="img/logo.jpg" alt="logo" id="logo"></a> </div> </div> <div class="top-bar-right"> <h3 class="hide-for-small">Signal Peptide Website</h3> <div id="responsive-menu"> <ul class="dropdown menu" data-dropdown-menu> <li><a href="./?m=myprotein">Search my Protein</a></li> <li><a href="./?m=searchspdb">Advanced Search</a></li> <li><a href="./?m=listspdb">Database Search</a></li> <li><a href="./?m=references">References</a></li> <li><a href="./?m=hints">Hints</a></li> <li><a href="./?m=links">Links</a></li> <li><a href="./?m=imprint">Imprint</a></li> </ul> </div> </div> </div> </div> <br> <div class="row columns"> <div class="content"> <span class="headline">Signal Peptide Database - Bacteria</span><br><br> <form action="index.php" method="post"><input type="hidden" name="sess" value=""> <input type="hidden" name="m" value="listspdb_bacteria"> <input type="hidden" name="id" value="2"> <input type="hidden" name="a" value="save"> <table cellspacing="2" cellpadding="2" border="0"> <tr> <td colspan="3" class="tabhead"> <b>Entry Details</b></td></tr> <tr height="23"> <td class="highlight">ID</td> <td class="highlight" width="50"> </td> <td class="highlight">2</td> </tr> <tr height="23"> <td class="highlight">Source Database</td> <td class="highlight" width="50"> </td> <td class="highlight">UniProtKB/Swiss-Prot</td> </tr> <tr height="23"> <td class="highlight">UniProtKB/Swiss-Prot Accession Number</td> <td class="highlight" width="50"> </td> <td class="highlight">A6X5T5    (Created: 2009-01-20 Updated: 2009-01-20)</td> </tr> <tr height="23"> <td class="highlight">UniProtKB/Swiss-Prot Entry Name</td> <td class="highlight" width="50"> </td> <td class="highlight"><a target="_new" class="bblack" href="http://www.uniprot.org/uniprot/14KL_OCHA4">14KL_OCHA4</a></td> </tr> <tr height="23"> <td class="highlight">Protein Name</td> <td class="highlight" width="50"> </td> <td class="highlight">Lectin-like protein BA14k</td> </tr> <tr height="23"> <td class="highlight">Gene</td> <td class="highlight" width="50"> </td> <td class="highlight">Oant_3884</td> </tr> <tr height="23"> <td class="highlight">Organism Scientific</td> <td class="highlight" width="50"> </td> <td class="highlight">Ochrobactrum anthropi (strain ATCC 49188 / DSM 6882 / NCTC 12168)</td> </tr> <tr height="23"> <td class="highlight">Organism Common</td> <td class="highlight" width="50"> </td> <td class="highlight"></td> </tr> <tr height="23"> <td class="highlight">Lineage</td> <td class="highlight" width="50"> </td> <td class="highlight">Bacteria<br>  Proteobacteria<br>    Alphaproteobacteria<br>      Rhizobiales<br>        Brucellaceae<br>          Ochrobactrum<br></td> </tr> <tr height="23"> <td class="highlight">Protein Length [aa]</td> <td class="highlight" width="50"> </td> <td class="highlight">151</td> </tr> <tr height="23"> <td class="highlight">Protein Mass [Da]</td> <td class="highlight" width="50"> </td> <td class="highlight">17666</td> </tr> <tr height="23"> <td class="highlight">Features</td> <td class="highlight" width="50"> </td> <td class="highlight"><table><tr><td><b>Type</b></td><td><b>Description</b></td><td><b>Status</b></td><td><b>Start</b></td><td><b>End</b></td></tr><tr><td class="w"><font color="red">signal peptide</font>   </td><td class="w"><font color="red"></font>   </td><td class="w"><font color="red">potential</font>   </td><td class="w"><font color="red">1</font>   </td><td class="w"><font color="red">26</font></td></tr><tr><td class="w"><font color="blue">chain</font>   </td><td class="w"><font color="blue">Lectin-like protein BA14k</font>   </td><td class="w"><font color="blue"></font>   </td><td class="w"><font color="blue">27</font>   </td><td class="w"><font color="blue">151</font></td></tr><tr><td class="w"><font color="green">transmembrane region</font>   </td><td class="w"><font color="green"></font>   </td><td class="w"><font color="green">potential</font>   </td><td class="w"><font color="green">83</font>   </td><td class="w"><font color="green">103</font></td></tr></table></td> </tr> <tr height="23"> <td class="highlight">SP Length</td> <td class="highlight" width="50"> </td> <td class="highlight">26</td> </tr> <tr valign="top"> <td class="highlight"></td><td class="highlight" width="50"> </td><td class="highlightfixed">----+----1----+----2----+----3----+----4----+----5</td></tr><tr valign="top"> <td class="highlight">Signal Peptide</td><td class="highlight" width="50"> </td><td class="highlightfixed">MNIFKQTCVGAFAVIFGATSIAPTMA</td></tr><tr valign="top"> <td class="highlight"> Sequence</td><td class="highlight" width="50"> </td><td class="highlightfixed"><font color="red">MNIFKQTCVGAFAVIFGATSIAPTMA</font><font color="blue">APLNLERPVINHNVEQVRDHRRPP<br>RHYNGHRPHRPGYWNGHRGYRHYRHGYRRYND</font><font color="green">GWWYPLAAFGAGAIIGGA<br>VSQ</font><font color="blue">PRPVYRAPRMSNAHVQWCYNRYKSYRSSDNTFQPYNGPRRQCYSPYS<br>R</td></tr><tr valign="top"> <td class="highlight"> Original</td><td class="highlight" width="50"> </td><td class="highlightfixed">MNIFKQTCVGAFAVIFGATSIAPTMAAPLNLERPVINHNVEQVRDHRRPP<br>RHYNGHRPHRPGYWNGHRGYRHYRHGYRRYNDGWWYPLAAFGAGAIIGGA<br>VSQPRPVYRAPRMSNAHVQWCYNRYKSYRSSDNTFQPYNGPRRQCYSPYS<br>R</td></tr><tr valign="top"> <td class="highlight"></td><td class="highlight" width="50"> </td><td class="highlightfixed">----+----1----+----2----+----3----+----4----+----5</td></tr><tr height="23"> <td class="highlight">Hydropathies</td> <td class="highlight" width="50"> </td> <td class="highlight"><a href="./hydropathy/hydropathy.php?id=2" target="_new"><img src="./hydropathy/hydropathy.php?id=2" border="0" width="600"></a></td> </tr> <tr> <td colspan="3" class="nohighlight"> </td> </tr> <tr> <td colspan="3" class="tabhead" align="center"><input class="button" type="reset" value="Back" onclick="history.back(-1);"></td> </tr> </table> </form></div> <hr> <div class="row"> <div class="small-4 medium-3 columns"><a href="./">Home</a>   <a href="./?m=imprint">Imprint</a></div> <div class="small-8 medium-9 columns text-right"> © 2007-2017 <a href="mailto:kapp@mpi-cbg.de">Katja Kapp</a>, Dresden & <a href="http://www.thpr.net/">thpr.net e. K.</a>, Dresden, Germany, last update 2010-06-11 </div> </div><br><br> <script src="js/vendor/jquery.js"></script> <script src="js/foundation.js"></script> <script> $(document).foundation(); </script> </body> </html> Process finished with exit code 0

所以看来这可以解决您的问题。

Answer 2

尝试一下：

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

url = 'http://www.signalpeptide.de/index.php?sess=&m=listspdb_bacteria&s=details&id=1000&listname='    
r = requests.get(url)

tabs = soup.find_all('table')
my_tab = pd.read_html(str(tabs[0]))
my_tab[0].drop(my_tab[0].columns[1], axis=1).drop(my_tab[0].index[0])

这应该在页面上以“ id 1000”开头输出主表。

Answer 3

如果您在汤中找到表，但在使用read_html进行解析时找不到表，则原因可能是特定的表已隐藏。因此，您可以使用以下快照：

import bs4
import pandas

# open file available at file_path
with open(file_path, encoding='utf-8') as fobj:
    soup = bs4.BeautifulSoup(fobj, 'html5lib')

# provide your table's class_name
tables = soup.find_all('table', attrs={'class': 'class_name'})


for table in tables:
    filtered_lines = list()
    data_frame = pandas.read_html(str(table), displayed_only=False)

注意：read_html中的display_only选项将允许您解析隐藏的表。

当我用beautifulSoup而不是Pandas刮时为什么会有桌子

3 个答案:

更新