Question

免责声明：这是我第一次涉足网络抓取活动

我有一个与搜索结果相对应的网址列表，例如

http://www.vinelink.com/vinelink/servlet/SubjectSearch?siteID=34003&agency=33&offenderID=2662

我正在尝试使用Selenium来访问结果的HTML，如下所示：

for url in detail_urls:
    driver.get(url)
    html = driver.page_source
    soup = BeautifulSoup(html, 'html.parser')
    print(soup.prettify())

但是，当我梳理所得的美化汤时，我注意到我需要的成分丢失了。回顾页面加载过程，我发现URL重定向了几次，如下所示：

有人对如何访问最终搜索结果数据有提示吗？

更新：经过进一步的探索，这似乎与正在执行的脚本有关，以检索相关数据以供显示...在page_source中引用了许多与搜索结果相关的脚本;有没有办法确定哪个相关？

我能够根据这张图片检查我需要的信息：

Answer 1

在将soup变量和HTML一起使用后，请遵循以下代码。

import json
data = soup.find('search-result')['data']
print(data)

输出：现在将每个值都当作字典。

{"offender_sid":154070373,"siteId":34003,"siteDesc":"NC_STATE","first_name":"WESLEY","last_name":"ADAMS","middle_initial":"CHURCHILL","alias_first_name":null,"alias_last_name":null,"alias_middle_initial":null,"oid":"2662","date_of_birth":"1965-11-21","agencyDesc":"Durham County Detention Center","age":53,"race":2,"raceDesc":"African American","gender":null,"genderDesc":null,"status_detail":"Durham County Detention Center","agency":33,"custody_status_cd":1,"custody_detail_cd":33,"custody_status_description":"In Custody","aliasFlag":false,"registerValid":true,"detailAgLink":false,"linkedCases":false,"registerMessage":"","juvenile_flg":0,"vineLinkInd":1,"vineLinkAgAccessCd":2,"links":[{"rel":"agency","href":"//www.vinelink.com/VineAppWebService/api/site/agency/34003/33"},{"rel":"self","href":"//www.vinelink.com/VineAppWebService/api/offender/?offSid=154070373&lang=en_US"}],"actions":[{"name":"register","template":"//www.vinelink.com/VineAppWebService/api/register/{json data}","method":"POST"}]}

下一步：

info = json.loads(data)

print(info['first_name'], info['last_name'])

#This prints the first and last name but you can get others, just get the key like 'date_of_birth' or 'siteId'. You can also assign them to variables.

Python（Selenium / BeautifulSoup）搜索结果动态网址

1 个答案: