Question

我有网址https://cars.mail.ru/reviews/renault/?year=2010-2016 我应该从那里打开

https://cars.mail.ru/reviews/renault/sandero_stepway/2015/143355/
https://cars.mail.ru/reviews/renault/sandero/2015/147850/
https://cars.mail.ru/reviews/renault/sandero/2012/147529/
https://cars.mail.ru/reviews/renault/duster/2014/147433/
https://cars.mail.ru/reviews/renault/logan/2011/146991/
https://cars.mail.ru/reviews/renault/duster/2015/146645/

我需要打开所有这些链接并在下一页旁边打开所有链接。我怎么能快速做到？如果我使用

models = ['11', '12', '14', '15', '16', '17', '18', '19', '20', '21', '25', '30', '4', '5', '6', '9',
     'avantime', 'clio', 'clio_rs', 'duster', 'espace', 'estafette', 'express', 'fluence',
     'fuego', 'grand_espace', 'grand_scenic', 'kangoo', 'kaptur', 'koleos', 'laguna', 'latitude',
     'logan', 'mascott', 'master', 'megane', 'megane_rs', 'modus', 'safrane', 'sandero', 'sandero_stepway',
     'scenic', 'symbol', 'trafic', 'twingo', 'vel_satis']
years = ['2010', '2011', '2012', '2013', '2014', '2015', '2016']
pattern = 'https://cars.mail.ru/reviews/renault/'

for model in models:
    for year in years:
        for i in range(143350, 143360):
            res = pattern + model + '/' + year + '/' + str(i)
            try:
                page = urllib2.urlopen(res).read()
                print page
                soup = BeautifulSoup(page, 'html.parser')
            except:
                continue

需要花费很多时间

Answer 1

你在做什么

./lib/index.js

HTTP请求。如果每个只需要一秒钟，你就会忙一个小时。

您可以尝试multiprocessing。

Python：从url打开一些引用

1 个答案: