Question

我是网络抓取的新手，我正在尝试使用python从此页面抓取特定内容：TRAC - People and Staffing

我需要获得的数据是列表中每个区域的律师的计数（甚至不需要姓名），无论是历史的还是最近的。

我用Google搜索并输入了美味的汤和dryscape。我已经阅读了几个教程，但我不了解javascript，所以我很难搞清楚哪个软件包最好，以及如何更改我找到的示例代码。

编辑：我没有订阅这个数据库，但是当我查看源代码时，我可以看到我需要的所有数据：

Screenshot

显示选择一个区和一个时间段时的名称列表;基本上，我需要的是一种自动迭代并存储输出的方法，按区域排序。

Answer 1

编辑：好吧，看起来我正在查看错误的子页面（评委，而不是检察官）。正确的页面是http://tracfed.syr.edu/express/fedstaf/initials/crimlist_pros.html

如果您查看页面来源，第76,82和85行，您会看到当您更新distric2选择框或period单选按钮时，它会调用名为{的javascript函数{1}}更改了检察官选择框的内容。

该功能在第189-206行中定义;一些整理和一些评论添加，它看起来像

updateAUSAMenu()

看一下，您可以看到生成的网址格式为

function updateAUSAMenu() {
    // get name of currently-selected district
    var DistName = document.forms[0].distric2.options[document.forms[0].distric2.selectedIndex].value;
    // clean up district name - remove commas, periods, and spaces
    DistName = replace(DistName, ',', '');
    DistName = replace(DistName, '.', '');
    DistName = replace(DistName, ' ', '');

    // decide which period is selected (recent or historical)
    var period = document.forms[0].period[0].checked;
    var insert;
    if (period==true) {
        insert= "";       // recent
    } else {
        insert="hist_";   // historical
    }

    // debug message
    // alert("./hidden_ausa/menu_ausa_" + insert + DistName + ".html");

    // build url
    var newurl = "./hidden_ausa/menu_ausa_" + insert + DistName + ".html" ;
    // load update script in a hidden frame
    // (loading the script runs it, and it then updates the prosecutor select-box)
    parent.frames['hidden'].location = newurl;
}

您可以直接从./hidden_ausa/menu_ausa_{district}.html # recent ./hidden_ausa/menu_ausa_hist_{district}.html # historical获取地区列表，如下所示：

http://tracfed.syr.edu/js/district_selector.js

然后我们可以得到像

这样的地区检察官

# assuming Python 3 with requests module:
import re
import requests

url       = "http://tracfed.syr.edu/js/district_selector.js"
js        = requests.get(url).content.decode()
districts = re.findall(r'<option value=\\"(.*?)\\">', js)

# looking at the values in `districts`, there are no periods or commas
#   or internal spaces; all we need to do is strip trailing spaces
districts = [d.strip() for d in districts]

给出了

from time import sleep

base     = "http://tracfed.syr.edu/express/fedstaf/initials"
rec_url  = base + "/hidden_ausa/menu_ausa_{district}.html"
hist_url = base + "/hidden_ausa/menu_ausa_hist_{district}.html"

# set up data structure
data = {
    district: {"recent":[], "historical":[]}
    for district in districts
}

# go get data!
for district in districts:
    # get recent data
    url  = rec_url.format(district=district)
    js   = requests.get(url).content.decode()
    pros = re.findall(r'Array\("(.*?)"', js)
    data[district]["recent"] = pros
    num_rec = len(pros)
    sleep(5)

    # get historical data
    url  = hist_url.format(district=district)
    js   = requests.get(url).content.decode()
    pros = re.findall(r'Array\("(.*?)"', js)
    data[district]["historical"] = pros
    num_hist = len(pros)
    sleep(5)

    # show progress
    print("{}: {} hist, {} rec".format(district, num_hist, num_rec), flush=True)

然后您需要弄清楚如何存储Ala_M: 40 hist, 16 rec Ala_N: 77 hist, 32 rec Ala_S: 32 hist, 15 rec Alaska: 34 hist, 21 rec # etc...以供进一步使用。

Answer 2

在大多数情况下，您不需要了解JavaScript来进行网页抓取。

打开您的开发者工具网络标签，然后尝试更改表单中的区域。您将看到您的浏览器对某些地址发出http请求：其中一个包含您需要的数据：

Developer Tools

浏览器对http://tracfed.syr.edu/express/fedstaf/initials/hidden_ausa/menu_ausa_Arizona.html这样的地址执行GET请求，并获得类似这样的响应：

<SCRIPT LANGUAGE="JavaScript">
<!--
var group= new Array() ;
   group[0] = new Array("Aguilar, Arturo","14006");
   group[1] = new Array("Albert, Jerry R.","00113");
   group[2] = new Array("Alexander, Bret S.","13950");
   group[3] = new Array("Anderson, Beverly K.","10220");
   group[4] = new Array("Arellano, Raquel","00114");
   group[5] = new Array("Bachus, Alison S.","13936");
   group[6] = new Array("Barry, Patrick T.","13937");
   group[7] = new Array("Battista, Frederick A.","00119");
   group[8] = new Array("Bibles, Camilla","00121");
   group[9] = new Array("Bostwick, Reese V.","00123");
   group[10] = new Array("Boyle, John Z.","10223");
   group[11] = new Array("Brown, Christopher A.","13982");
   group[12] = new Array("Bullis, Paul","14005");
   group[13] = new Array("Cabanillas, Cristina M.","00126");
   group[14] = new Array("Cassell, Matthew Colson","14017");
   group[15] = new Array("Cerow, Darcy A.","00127");
   group[16] = new Array("Christiansen, Callie","14010");
   group[17] = new Array("Clausen, Monte C.","00132");
   group[18] = new Array("Clemens, Shelley K. G.","13931");
   group[19] = new Array("Cocio, Fredrick A.","13945");
   group[20] = new Array("Corbin, Carmen","13996");
   group[21] = new Array("Coughlin, Robert E.","14022");
   group[22] = new Array("Davenport, Gordon","13985");
   group[23] = new Array("DeJong, Sarah Elizabeth","14014");
   group[24] = new Array("Decosta, Sabrina","13952");
   group[25] = new Array("Dejoe, Ryan","14001");
   group[26] = new Array("Demarais, Ann L.","13962");
   group[27] = new Array("Dokken, Roger W.","00135");
   group[28] = new Array("Duryee, Carin","00138");
   group[29] = new Array("Eltringham, Matthew G.","14019");
   group[30] = new Array("Evans, John Reynolds","05657");
   group[31] = new Array("Feldmeier, Mary Sue","00142");
   group[32] = new Array("Fellrath, Robert A.","13981");
   group[33] = new Array("Ferg, Bruce M.","10234");
   group[34] = new Array("Ferraro, D. Thomas","10236");
   group[35] = new Array("Figueroa, Jesse J.","00143");
   group[36] = new Array("Fisher, Joshua Ivan","14012");
   group[37] = new Array("Flannigan, David P","10237");
   group[38] = new Array("Fugitive","00141");
   group[39] = new Array("Galati, Frank T.","13927");
   group[40] = new Array("Galbraith, Charles","13998");
   group[41] = new Array("Granoff, Jonathan B.","10241");
   group[42] = new Array("Granoff, Liza M.","13941");
   group[43] = new Array("Green, Jennifer E.","13954");
   group[44] = new Array("Greer, Dyanne C.","10242");
   group[45] = new Array("Hanley, Joseph","13997");
   group[46] = new Array("Hansen, Sandra M.","00153");
   group[47] = new Array("Healey, Kyle","13994");
   group[48] = new Array("Hernandez, Rachel","00155");
   group[49] = new Array("Hodahkwen, Marnie","14002");
   group[50] = new Array("Hollon, Leta","13971");
   group[51] = new Array("Hopkins, Kimberly E.","13989");
   group[52] = new Array("Howe, Randall","13969");
   group[53] = new Array("Hurley, Emory T.","10248");
   group[54] = new Array("Hyder, Charles F.","00159");
   group[55] = new Array("Jennis, Lisa","00161");
   group[56] = new Array("Kasprzyk, Brian","13976");
   group[57] = new Array("Kelly, Kristen","14003");
   group[58] = new Array("Kern, David A.","00166");
   group[59] = new Array("Kim, Wendy","13991");
   group[60] = new Array("Kimmins, Lynnette","00167");
   group[61] = new Array("Kirby, Vincent Q.","00168");
   group[62] = new Array("Kleindienst, Wallace H.","00170");
   group[63] = new Array("Kleiner, Albert L.","10255");
   group[64] = new Array("Knapp, James Richard","13940");
   group[65] = new Array("Koehler, Joseph E.","00172");
   group[66] = new Array("Kokanovich, Mark Samuel","14021");
   group[67] = new Array("Lacey, James T.","00174");
   group[68] = new Array("Landis, Brent H.","13968");
   group[69] = new Array("Langhofer, Kory","13992");
   group[70] = new Array("Lanham, Krissa","14009");
   group[71] = new Array("Lanza, Dominc","13984");
   group[72] = new Array("Laramore, Stephen W.","00175");
   group[73] = new Array("Larson, Brian","00176");
   group[74] = new Array("Lee, Lawrence C.","13987");
   group[75] = new Array("Lee, Michael A.","10258");
   group[76] = new Array("Lefkowitz, Claire K.","00177");
   group[77] = new Array("Lemke, Kathy","13951");
   group[78] = new Array("Levinson, Jennifer F.","13964");
   group[79] = new Array("Lewis, Christopher J.","13963");
   group[80] = new Array("Lodge, Joseph J.","00180");
   group[81] = new Array("Logalbo, Michael D.","13932");
   group[82] = new Array("Logan, Steven P.","00181");
   group[83] = new Array("Lopez, John R., IV","10261");
   group[84] = new Array("Lucca, Jonell L.","13965");
   group[85] = new Array("Maingot, Anthony Edward","13939");
   group[86] = new Array("Markovick, Erick","00183");
   group[87] = new Array("Marlowe, Joelyn D.","00184");
   group[88] = new Array("McCallum, Erica","13973");
   group[89] = new Array("McCormick, Glenn B.","10267");
   group[90] = new Array("McDonald, Karen S.","00190");
   group[91] = new Array("McGhee, James","14007");
   group[92] = new Array("McGhee, Melanie A.","14018");
   group[93] = new Array("McLaughlin, Jane E.","13960");
   group[94] = new Array("McMurray, Molly","14008");
   group[95] = new Array("Meister, Melissa","14004");
   group[96] = new Array("Melissa, Karlen","13978");
   group[97] = new Array("Mellor, Joshua C.","13988");
   group[98] = new Array("Miskell, Robert L.","00193");
   group[99] = new Array("Morrissey, Michael T.","00194");
   group[100] = new Array("Morse, James B.","13953");
   group[101] = new Array("Mosher, Anne E.","00195");
   group[102] = new Array("Nelson, Heather H.","13957");
   group[103] = new Array("Novitsky, Sharon K.","00198");
   group[104] = new Array("Parecki, Josh P.","13958");
   group[105] = new Array("Passos, Raynette","00201");
   group[106] = new Array("Peery, Amy Elizabeth","14020");
   group[107] = new Array("Perkel, Walter P","13977");
   group[108] = new Array("Petermann, David P.","10272");
   group[109] = new Array("Phillips, Sheila","13979");
   group[110] = new Array("Picton, Cory","13972");
   group[111] = new Array("Pimsner, David","00203");
   group[112] = new Array("Pop, Anca","13999");
   group[113] = new Array("Rapp, Kevin M.","00206");
   group[114] = new Array("Rassas, Theresa","13970");
   group[115] = new Array("Reid-Moore, Christina","13990");
   group[116] = new Array("Restaino, Gary M.","10274");
   group[117] = new Array("Rocker, Wilbert L., Jr.","13959");
   group[118] = new Array("Roetzel, Danny N.","10276");
   group[119] = new Array("Rolley, Karen","14000");
   group[120] = new Array("Rood, Paul V.","00209");
   group[121] = new Array("Russell, Craig Howard","14016");
   group[122] = new Array("Sampson, Dimitra H.","13967");
   group[123] = new Array("Sardelli, Brian G.","14015");
   group[124] = new Array("Savel, Nicole P.","10277");
   group[125] = new Array("Scheel, Ann B.","00212");
   group[126] = new Array("Schmit, Gerard Micah","13956");
   group[127] = new Array("Schneider, Patrick J.","00213");
   group[128] = new Array("Sexton, Peter S.","00214");
   group[129] = new Array("Sharda, Munish","10278");
   group[130] = new Array("Silver, James Anthony","13949");
   group[131] = new Array("Simon, Thomas C.","00216");
   group[132] = new Array("Spaven, Michelle","13995");
   group[133] = new Array("Strange, Elizabeth A.","13961");
   group[134] = new Array("Sukenic, Howard D.","10283");
   group[135] = new Array("Timm, Craig M.","13983");
   group[136] = new Array("Tsethlikai, Serra","00220");
   group[137] = new Array("Uhl, Louis","13974");
   group[138] = new Array("Van Buskirk, Tracy","13986");
   group[139] = new Array("Vercauteren, Keith E.","10285");
   group[140] = new Array("Walsh, Janet M.","10286");
   group[141] = new Array("Wang, Rui","13993");
   group[142] = new Array("Wiles, Jennifer Marie","14011");
   group[143] = new Array("Woo, Cassie","13975");
   group[144] = new Array("Woo, Raymond K.","13943");
   group[145] = new Array("Woolridge, Angela W.","13942");
   group[146] = new Array("Zipps, David R.","13980");
parent.frames['form'].changeAUSAMenu(group); 
delete group ; 
//-->
</SCRIPT>
<HTML><BODY BGCOLOR="#F5F6C8"></BODY></HTML>

现在肯定是开发人员选择响应AJAX请求的奇怪方式（通常网页上的客户端 - 服务器交互是使用JSON / XML完成的），但是我们可以用不那么复杂的正则表达式来解析数据（I＆＃ 39;请留下给你以弄清楚如何编写一个从答复文本中提取律师姓名的人。）

如何使用python从具有多个动态选择字段的页面中抓取数据？

2 个答案: