Question

我使用python selenium自动化脚本以从站点获取数据并生成为列表。现在从子列表的输出我只需要获得第二个元素。请参考下面生成列表的代码

# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
from datetime import datetime
import dateutil.parser
import urllib2
import requests
import htmllib
import sys
import re
import csv
import pandas as pd
from operator import itemgetter
from selenium import webdriver
chrome_path = r"/usr/bin/chromedriver"
driver = webdriver.Chrome(chrome_path)
driver.get("http://windspeed.atcouncil.org/")  # opening the site
driver.find_element_by_xpath(
    """//*[@id="address"]""").click()  # click the radio button
driver.find_element_by_xpath("""//*[@id="google-map-address"]""").click()  # clicking the textbox
cities = ['pheonix']  # city list
for city in cities:
    # print (city)
    driver.find_element_by_xpath("""//*[@id="google-map-address"]""").send_keys(city)  # passing cities
    driver.find_element_by_xpath("""//*[@id="searchform"]/div[1]/div[2]/button""").click()
    driver.find_element_by_xpath("""// *[ @ id = "latt"]""")
    driver.find_element_by_xpath('//*[@id="searchform"]/div[1]/div[7]/span/input').click()
x = driver.current_url
#print x
data = []
newList = []
latest = []
driver.get(x)
s = driver.find_element_by_id('bodyContent')  #search results div
data = '\n'.join(s.text.split('\n')[3:22])
data = str(data).split("\n")
#for each in data:
    #data = data[each].split(':',1)[1]
#print data
#data= [i.split('\t', 1)[0] for i in data]
for i in data:
    newList.append(i.split(':'))
print data
print newList


#[latest.append(x[1]) for x in newList]
#b = map(itemgetter(1), newList)
#print b
if type(newList) is list:
    print 'list'
else:
    print 'none'

我得到的输出是

['Query Date: Wed Aug 16 2017', 'Latitude: 33.4484', 'Longitude: -112.0740', '', 'ASCE 7-10 Windspeeds', '(3-sec peak gust in mph*):', '', 'Risk Category I: 105', 'Risk Category II: 115', 'Risk Category III-IV: 120', 'MRI** 10-Year: 76', 'MRI** 25-Year: 84', 'MRI** 50-Year: 90', 'MRI** 100-Year: 96', '', 'ASCE 7-05 Windspeed:', '  90 (3-sec peak gust in mph)', 'ASCE 7-93 Windspeed:', '  72 (fastest mile in mph)']
[['Query Date', ' Wed Aug 16 2017'], ['Latitude', ' 33.4484'], ['Longitude', ' -112.0740'], [''], ['ASCE 7-10 Windspeeds'], ['(3-sec peak gust in mph*)', ''], [''], ['Risk Category I', ' 105'], ['Risk Category II', ' 115'], ['Risk Category III-IV', ' 120'], ['MRI** 10-Year', ' 76'], ['MRI** 25-Year', ' 84'], ['MRI** 50-Year', ' 90'], ['MRI** 100-Year', ' 96'], [''], ['ASCE 7-05 Windspeed', ''], ['  90 (3-sec peak gust in mph)'], ['ASCE 7-93 Windspeed', ''], ['  72 (fastest mile in mph)']]
list

从上面的输出我想得到第二个元素，如下所示，

['Wed Aug 16 2017', '33.4484', '-112.0740', '105', ' 115', ' 120',' 76', ' 84', ' 90',' 96'   '90 (3-sec peak gust in mph)',  '72 (fastest mile in mph)']

Answer 1

您要查找的信息不在任何标记下。它只是bodyContent标记的一部分。但是每个分隔都在HTML中创建一个文本节点。因此，节点上的文本节点可以使用/text() XPATH。

<div id="bodyContent"><h1><b>Search Results</b></h1><br><br><b>Query Date: </b> Wed Aug 16 2017<br><b>Latitude: </b> 33.4484<br><b>Longitude: </b> -112.0740<br><br><b>ASCE 7-10 Windspeeds <br>(3-sec peak gust in mph*):</b><br><br><b>Risk Category I:</b> 105<br><b>Risk Category II:</b> 115<br><b>Risk Category III-IV:</b> 120<br><b>MRI** 10-Year:</b> 76<br><b>MRI** 25-Year:</b> 84<br><b>MRI** 50-Year:</b> 90<br><b>MRI** 100-Year:</b> 96<br><br><b>ASCE 7-05 Windspeed:</b><br> &nbsp; 90 (3-sec peak gust in mph)<br><b>ASCE 7-93 Windspeed:</b><br> &nbsp; 72 (fastest mile in mph)<br><br><p></p><br><p>*Miles per hour<br>**Mean Recurrence Interval</p><br><p>Users should consult with local building officials <br> to determine if there are community-specific wind speed <br> requirements that govern.</p><br><form id="createPDF" action="/pdf/create.php" method="post"><input type="hidden" name="cont" value="<h1><b>Search Results</b></h1><br/><br/><b>Query Date: </b> Wed Aug 16 2017<br/><b>Latitude: </b> 33.4484<br/><b>Longitude: </b> -112.0740<br/><br/><b>ASCE 7-10 Windspeeds <br/>(3-sec peak gust in mph*):</b><br/><br/><b>Risk Category I:</b> 105<br/><b>Risk Category II:</b> 115<br/><b>Risk Category III-IV:</b> 120<br/><b>MRI** 10-Year:</b> 76<br/><b>MRI** 25-Year:</b> 84<br/><b>MRI** 50-Year:</b> 90<br/><b>MRI** 100-Year:</b> 96<br/><br/><b>ASCE 7-05 Windspeed:</b><br/> &nbsp; 90 (3-sec peak gust in mph)<br/><b>ASCE 7-93 Windspeed:</b><br/> &nbsp; 72 (fastest mile in mph)<br/><br/><p></p><br/><p>*Miles per hour<br/>**Mean Recurrence Interval</p><br/><p>Users should consult with local building officials <br/> to determine if there are community-specific wind speed <br/> requirements that govern.</p><br/>"><input type="hidden" name="lat" value="33.4484"><input type="hidden" name="lan" value="-112.0740"><input type="hidden" name="zoom" id="google-map-zoom" value="4"><input type="hidden" name="maptype" id="google-map-maptype" value="roadmap"><!-- <a href="#" onclick="document.getElementById('createPDF').submit(); return false;"><img src="/images/pdf.png" border=0 /> Download a PDF of your results</a> --></form><br><a href="#" onclick="window.print(); return false;"><img src="/images/print.png" border="0"> Print your results</a><br><br></div>

现在大多数抓取框架或html解析都允许您提取节点数据，但Selenium总是对元素感兴趣。所以它不能自己提取东西。你需要写一些相同的JavaScript。可以使用document.evaluate方法运行XPath并获取其所有结果。相同的javascript将如下

script_extract_data="""
var data = document.evaluate("./text()", 
             document.getElementById("bodyContent"), null, 
             XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);

var text = [];

for (var i=0; i < data.snapshotLength; i++) 
{
   text.push(data.snapshotItem(i).textContent.trim());
}

return text;
"""

您可以使用

执行脚本

data = driver.execute_script(script_extract_data)
print(data)

将生成以下输出

In[4]: driver.execute_script(script_extract_data)
Out[4]: 
['Wed Aug 16 2017',
 '33.4484',
 '-112.0740',
 '105',
 '115',
 '120',
 '76',
 '84',
 '90',
 '96',
 '90 (3-sec peak gust in mph)',
 '72 (fastest mile in mph)']

如何使用python从子列表中获取第二个元素

1 个答案: