如何让rp和xrange动态化?

时间:2013-06-02 04:42:28

标签: python xml dynamic web-scraping xrange

嘿伙计们非常感谢您抽出时间来查看我的问题,我一直在研究这段代码大约1周(我是新编码和1周的python)目前循环只适用于x in xrange (x)和'rp':'x'是此xml中可用的正确行数。 xml全天更新,我想知道是否有人可以提供使x动态的解决方案?

import mechanize
import urllib
import json
import re
from sched import scheduler
from time import time, sleep

from sched import scheduler
from time import time, sleep

s = scheduler(time, sleep)

def run_periodically(start, end, interval, func):
event_time = start
while event_time < end:
    s.enterabs(event_time, 0, func, ())
    event_time += interval
s.run()

def getData():  
post_url = "urlofinterest_xml"
browser = mechanize.Browser()
browser.set_handle_robots(False)
browser.addheaders = [('User-agent', 'Firefox')]

######These are the parameters you've got from checking with the aforementioned tools
parameters = {'page' : '1',
              'rp' : '8',
              'sortname' : 'roi',
              'sortorder' : 'desc'
             }
#####Encode the parameters
data = urllib.urlencode(parameters)
trans_array = browser.open(post_url,data).read().decode('UTF-8')

xmlload1 = json.loads(trans_array)
pattern1 = re.compile('>&nbsp;&nbsp;(.*)<')
pattern2 = re.compile('/control/profile/view/(.*)\' title=')
pattern3 = re.compile('<span style=\'font-size:12px;\'>(.*)<\/span>')
pattern4 = re.compile('title=\'Naps posted: (.*) Winners:')
pattern5 = re.compile('Winners: (.*)\'><img src=')


for i in xrange(8):
    user_delimiter = xmlload1['rows'][i]['cell']['username']
    selection_delimiter = xmlload1['rows'][i]['cell']['race_horse']

    username_delimiter_results = re.findall(pattern1, user_delimiter)[0]
    userid_delimiter_results = int(re.findall(pattern2, user_delimiter)[0])
    user_selection = re.findall(pattern3, selection_delimiter)[0]
    user_numberofselections = float(re.findall(pattern4, user_delimiter)[0])
    user_numberofwinners = float(re.findall(pattern5, user_delimiter)[0])

    strikeratecalc1 = user_numberofwinners/user_numberofselections
    strikeratecalc2 = strikeratecalc1*100

    print "user id = ",userid_delimiter_results
    print "username = ",username_delimiter_results
    print "user selection = ",user_selection
    print "best price available as decimal = ",xmlload1['rows'][i]['cell']     ['tws.best_price']
    print "race time = ",xmlload1['rows'][i]['cell']['race_time']
    print "race meeting = ",xmlload1['rows'][i]['cell']['race_meeting']
    print "ROI = ",xmlload1['rows'][i]['cell']['roi']
    print "number of selections = ",user_numberofselections
    print "number of winners = ",user_numberofwinners
    print "Strike rate = ",strikeratecalc2,"%"
    print ""


getData()


run_periodically(time()+5, time()+1000000, 15, getData)

亲切的问候AEA

2 个答案:

答案 0 :(得分:4)

首先,我将谈谈你如何迭代你的结果。根据您的代码,xmlload1['rows']是一个dicts数组,因此您可以直接迭代它,而不是选择任意数字。为了使这个更好的例子,我将设置一些任意数据来明确这一点:

xmlload1 = {
   "rows": [{"cell": {"username": "one", "race_horse":"b"}}, {"cell": {"username": "two", "race_horse": "c"}}]
}

因此,根据上面的数据,您可以在for循环中迭代遍历行,如下所示:

for row in xmlload1['rows']:
    cell = row["cell"]
    print cell["username"]
    print cell["race_horse"]

每次迭代时,cell都会获取iterable中另一个元素的值(xmlload1['rows']中的列表)。这适用于支持迭代的任何容器或序列(如liststuplesdictsgenerators等。)

请注意我在任何地方都没有使用任何魔术数字,因此xmlload1['rows']可以任意长,并且仍可以使用。

您可以使用函数将请求设置为动态,如下所示:

def get_data(rp=8, page=1):
    parameters = {'page' : str(page),
              'rp' : str(rp),
              'sortname' : 'roi',
              'sortorder' : 'desc'
             }
    data = urllib.urlencode(parameters)
    trans_array = browser.open(post_url,data).read().decode('UTF-8')
    return json.loads(trans_array)

现在,您可以致电get_data(rp=5)获取5行,或get_data(rp=8)获取8行[和get_data(rp=8, page=3)获取第3页]等等。您可以明确添加其他内容变量甚至直接将parameters dict传递给函数。

答案 1 :(得分:2)

我不确定我理解你的问题,但我认为你想要的是:

rows = xmlload1['rows']
for row in rows:
    user_delimiter = row['cell']['username']
    selection_delimiter = row['cell']['race_horse']
    # ...

如果您需要行索引以及行本身,请使用enumerate

rows = xmlload1['rows']
for i, row in enumerate(rows):
    user_delimiter = row['cell']['username']
    selection_delimiter = row['cell']['race_horse']
    # ...

一般情况下,如果你为了除固定次数的迭代以外的任何目的而进行for i in range(…),那么你做错了。通常有一个你想要迭代的集合;找到它并迭代它。