Question

我正试图刮table，但我遇到了一些障碍。我想确保每个标题下的数据（例如<tr> <td> <table class="EPSBResultGrid" cellspacing="0" rules="cols" border="1" style="border-color:DarkGray;border-collapse:collapse;"> <tbody><tr class="EPSBResultGridHeader"> <th scope="col">Cred</th><th scope="col">Description</th><th scope="col">Effective</th><th scope="col">Expiration</th><th scope="col">Restricted To</th> </tr><tr class="EPSBResultGridHeader"> <td colspan="9" style="border-width:1px;border-style:solid;font-weight:bold;">Do Not Print (00)</td> </tr><tr class="EPSBResultGridItem"> <td>RANK1</td><td>Rank I</td><td>07<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl03_EFF_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl03_EFF_DATE_txtDateMM" value="07">-01<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl03_EFF_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl03_EFF_DATE_txtDateDD" value="01">-2000<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl03_EFF_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl03_EFF_DATE_txtDateYYYY" value="2000">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td>06<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl03_EXP_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl03_EXP_DATE_txtDateMM" value="06">-30<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl03_EXP_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl03_EXP_DATE_txtDateDD" value="30">-2020<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl03_EXP_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl03_EXP_DATE_txtDateYYYY" value="2020">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td></td> </tr><tr class="EPSBResultGridHeader"> <td colspan="9" style="border-width:1px;border-style:solid;font-weight:bold;">Cert Issued (30)</td> </tr><tr class="EPSBResultGridAlternatingItem"> <td>G20</td><td>Middle School Teaching Field: Social Studies</td><td>07<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl05_EFF_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl05_EFF_DATE_txtDateMM" value="07">-01<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl05_EFF_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl05_EFF_DATE_txtDateDD" value="01">-1995<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl05_EFF_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl05_EFF_DATE_txtDateYYYY" value="1995">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td>06<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl05_EXP_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl05_EXP_DATE_txtDateMM" value="06">-30<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl05_EXP_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl05_EXP_DATE_txtDateDD" value="30">-2020<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl05_EXP_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl05_EXP_DATE_txtDateYYYY" value="2020">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td></td> </tr><tr class="EPSBResultGridItem"> <td>G71</td><td>Middle School Teaching Field: Mathematics</td><td>07<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl06_EFF_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl06_EFF_DATE_txtDateMM" value="07">-01<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl06_EFF_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl06_EFF_DATE_txtDateDD" value="01">-1995<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl06_EFF_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl06_EFF_DATE_txtDateYYYY" value="1995">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td>06<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl06_EXP_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl06_EXP_DATE_txtDateMM" value="06">-30<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl06_EXP_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl06_EXP_DATE_txtDateDD" value="30">-2020<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl06_EXP_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl06_EXP_DATE_txtDateYYYY" value="2020">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td></td> </tr><tr class="EPSBResultGridAlternatingItem"> <td>PCS</td><td>Provisional Certificate For Guidance Counselor, Secondary Grades 5-12</td><td>01<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl07_EFF_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl07_EFF_DATE_txtDateMM" value="01">-01<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl07_EFF_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl07_EFF_DATE_txtDateDD" value="01">-2016<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl07_EFF_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl07_EFF_DATE_txtDateYYYY" value="2016">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td>06<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl07_EXP_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl07_EXP_DATE_txtDateMM" value="06">-30<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl07_EXP_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl07_EXP_DATE_txtDateDD" value="30">-2020<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl07_EXP_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl07_EXP_DATE_txtDateYYYY" value="2020">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td></td> </tr><tr class="EPSBResultGridItem"> <td>PMBF</td><td>Provisional Certificate For Teaching In The Middle Grades 5-8 (And For Other Assignments As Identified By Kentucky Program Of Studies)</td><td>07<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl08_EFF_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl08_EFF_DATE_txtDateMM" value="07">-01<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl08_EFF_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl08_EFF_DATE_txtDateDD" value="01">-2015<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl08_EFF_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl08_EFF_DATE_txtDateYYYY" value="2015">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td>06<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl08_EXP_DATE_txtDateMM" name="ctl00_ContentPlaceHolder1_ctl00_ctl08_EXP_DATE_txtDateMM" value="06">-30<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl08_EXP_DATE_txtDateDD" name="ctl00_ContentPlaceHolder1_ctl00_ctl08_EXP_DATE_txtDateDD" value="30">-2020<input type="hidden" id="ctl00_ContentPlaceHolder1_ctl00_ctl08_EXP_DATE_txtDateYYYY" name="ctl00_ContentPlaceHolder1_ctl00_ctl08_EXP_DATE_txtDateYYYY" value="2020">You must enter a day.You must enter a month.You must enter a year.Invalid Date.</td><td></td> </tr> </tbody></table> Note: Suspended and revoked credentials are shown with red text with a strike through line. </td> </tr>）与相应的标题组合在一起。

当我尝试使用下面的html时出现问题。

class="EPSBResultGridItem"

标题（class="EPSBResultGridAlternatingItem"）下方的信息（class="EPSBResultHeader"＆amp; count = 0 header = tree.xpath( './/table/tr[@class="EPSBResultGridHeader"]') difference = 10 - len(header) for i in range(0, difference): header.append('') for license_row in header: count = count + 1 try: header_data = license_row.xpath(".//text()") header_data = clean(header_data) nested_data = license_row.xpath(".//following-sibling::tr//text()") nested_data = clean(nested_data) print count, header_data print count, nested_data except AttributeError: header_data = '' # Append licensure data if count == 1: lheader1.append(header_data) lheader_info1(nested_data) if count == 2: lheader2.append(header_data) lheader_info2(nested_data) if count == 3: lheader3.append(header_data) lheader_info3(nested_data) if count == 4: lheader4.append(header_data) lheader_info4(nested_data) if count == 5: lheader5.append(header_data) lheader_info5(nested_data)）未嵌套在其下方，因此我一直无法找到制作方法确保每个标题下的信息都与正确的标题组合在一起。

这是我的代码：

>>>print lheader_info2
['RANK1', 'Rank I', '07-01-2018', '06-30-2021']

>>>print lheader_info3
['G20', 'Middle School Teaching Field: Social Studies----', 'G30', 'Middle School Teaching Field: English And Communications----', 'ILE2', 'Professional Certificate For Instructional Leadership -- Early Elementary School Principal, Grades K-4; Level II', '07-01-2017', '06-30-2021', 'ILM2', 'Professional Certificate For Instructional Leadership--Middle Grade School Principal, Grades 5-8; Level II', '07-01-2017', '06-30-2021', 'ILV2', 'Professional Certificate For Instructional Leadership--Supervisor Of Instruction, Grades K-12; Level II', '07-01-2018', '06-30-2021', 'PMBF', 'Provisional Certificate For Teaching In The Middle Grades 5-8 (And For Other Assignments As Identified By Kentucky Program Of Studies)', '07-01-2016', '06-30-2021']

我的最终目标是拥有这样的输出：

lxml

我正在使用BeautifulSoup，但如果这似乎是一种更好的方法，我也使用了tearDown()。

Answer 1

我会找到每个子标题，并在满足另一个标题后，在next tr siblings遍历循环，或者到达表的末尾：

from collections import defaultdict
from pprint import pprint

import requests
from bs4 import BeautifulSoup

url = "https://wd.kyepsb.net/EPSB.WebApps/KECI/view_data.aspx?id=37161"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")

data = defaultdict(list)

table = soup.find("table", class_="EPSBResultGrid")
for header in table.select("tr.EPSBResultGridHeader")[1:]:
    header_name = header.get_text(strip=True)
    for row in header.find_next_siblings("tr"):
        if "EPSBResultGridHeader" in row.get("class", []):
            break

        data[header_name].append(row.td.get_text(strip=True))

pprint(dict(data))

打印：

{'Cert Issued (30)': ['G20', 'G30', 'ILE2', 'ILM2', 'ILV2', 'PMBF'],
 'Do Not Print (00)': ['RANK1'],
 'History (97)': ['ILE2', 'ILM2', 'ILV2', 'RANK1']}

这里的[1:]切片是跳过初始顶级表标题。

Answer 2

使用 lxml

def pair():
    tree = html.fromstring(requests.get(url).content)
    # get table and iterate over the trs
    iter_trs = tree.cssselect("table.EPSBResultGrid")[0].iter("tr")
    # skip the initial tr
    next(iter_trs)
    # first EPSBResultGridHeader
    start = next(iter_trs).xpath("td//text()")[0]
    nodes, tmp = {}, []
    # iterate over the rest of the nodes
    for node in iter_trs:
        # if we find another EPSBResultGridHeader, yield what we have and start again.
        if node.get("class") == "EPSBResultGridHeader":
            nodes[start] = tmp
            start, tmp = node.xpath("td//text()")[0], []
        else:
            tmp.append([td.xpath("normalize-space(.)").replace(ur"\xa0", "") for td in node.xpath("./td")])
    return nodes

哪会给你：

{'Cert Issued (30)': [[u'G20',
                       u'Middle School Teaching Field: Social Studies',
                       u'--',
                       u'--',
                       u''],
                      [u'G30',
                       u'Middle School Teaching Field: English And Communications',
                       u'--',
                       u'--',
                       u''],
                      [u'ILE2',
                       u'Professional Certificate For Instructional Leadership -- Early Elementary School Principal, Grades K-4; Level II',
                       u'07-01-2017',
                       u'06-30-2021',
                       u''],
                      [u'ILM2',
                       u'Professional Certificate For Instructional Leadership--Middle Grade School Principal, Grades 5-8; Level II',
                       u'07-01-2017',
                       u'06-30-2021',
                       u''],
                      [u'ILV2',
                       u'Professional Certificate For Instructional Leadership--Supervisor Of Instruction, Grades K-12; Level II',
                       u'07-01-2018',
                       u'06-30-2021',
                       u''],
                      [u'PMBF',
                       u'Provisional Certificate For Teaching In The Middle Grades 5-8 (And For Other Assignments As Identified By Kentucky Program Of Studies)',
                       u'07-01-2016',
                       u'06-30-2021',
                       u'']],
 'Do Not Print (00)': [[u'RANK1',
                        u'Rank I',
                        u'07-01-2018',
                        u'06-30-2021',
                        u'']],
 'History (97)': [[u'ILE2',
                   u'Professional Certificate For Instructional Leadership -- Early Elementary School Principal, Grades K-4; Level II',
                   u'07-01-2012',
                   u'06-30-2017',
                   u''],
                  [u'ILM2',
                   u'Professional Certificate For Instructional Leadership--Middle Grade School Principal, Grades 5-8; Level II',
                   u'07-01-2012',
                   u'06-30-2017',
                   u''],
                  [u'ILV2',
                   u'Professional Certificate For Instructional Leadership--Supervisor Of Instruction, Grades K-12; Level II',
                   u'07-01-2013',
                   u'06-30-2018',
                   u''],
                  [u'RANK1', u'Rank I', u'12-15-1995', u'06-30-2018', u'']]}

如果您想要平面列表，请使用扩展tmp.extend(td.xpath("normalize-space(.)").replace(u"\xa0", "") for td in node.xpath("./td")):

{'Cert Issued (30)': [u'G20',
                      u'Middle School Teaching Field: Social Studies',
                      u'--',
                      u'--',
                      u'',
                      u'G30',
                      u'Middle School Teaching Field: English And Communications',
                      u'--',
                      u'--',
                      u'',
                      u'ILE2',
                      u'Professional Certificate For Instructional Leadership -- Early Elementary School Principal, Grades K-4; Level II',
                      u'07-01-2017',
                      u'06-30-2021',
                      u'',
                      u'ILM2',
                      u'Professional Certificate For Instructional Leadership--Middle Grade School Principal, Grades 5-8; Level II',
                      u'07-01-2017',
                      u'06-30-2021',
                      u'',
                      u'ILV2',
                      u'Professional Certificate For Instructional Leadership--Supervisor Of Instruction, Grades K-12; Level II',
                      u'07-01-2018',
                      u'06-30-2021',
                      u'',
                      u'PMBF',
                      u'Provisional Certificate For Teaching In The Middle Grades 5-8 (And For Other Assignments As Identified By Kentucky Program Of Studies)',
                      u'07-01-2016',
                      u'06-30-2021',
                      u''],
 'Do Not Print (00)': [u'RANK1', u'Rank I', u'07-01-2018', u'06-30-2021', u''],
 'History (97)': [u'ILE2',
                  u'Professional Certificate For Instructional Leadership -- Early Elementary School Principal, Grades K-4; Level II',
                  u'07-01-2012',
                  u'06-30-2017',
                  u'',
                  u'ILM2',
                  u'Professional Certificate For Instructional Leadership--Middle Grade School Principal, Grades 5-8; Level II',
                  u'07-01-2012',
                  u'06-30-2017',
                  u'',
                  u'ILV2',
                  u'Professional Certificate For Instructional Leadership--Supervisor Of Instruction, Grades K-12; Level II',
                  u'07-01-2013',
                  u'06-30-2018',
                  u'',
                  u'RANK1',
                  u'Rank I',
                  u'12-15-1995',
                  u'06-30-2018',
                  u'']}

当行的html标记没有嵌套在标题标记下时，在表标题下面删除信息行

2 个答案: