Question

我用python结合selenium编写了一个脚本，利用代理来解析不同站点的标题。如果我坚持为每个站点创建单独的驱动程序实例，我的脚本就可以做到。但是，我的目标是无论遍历多少站点，都使用相同的浏览器（使用不同的代理）。

如何使用具有不同代理的同一浏览器来遍历多个站点以解析它们的标题？

我到目前为止的尝试：

import random
from itertools import cycle
from selenium import webdriver

proxyList = ['103.110.37.244:36022', '180.254.218.229:8080', '110.74.197.207:50632', '1.20.101.95:49001', '200.10.193.90:8080', '173.164.26.117:3128', '103.228.118.66:43002', '178.128.231.201:3128', '1.2.169.54:55312', '181.52.85.249:31487', '97.64.135.4:8080', '190.96.214.123:53251', '52.144.107.142:31923', '45.5.224.145:52035', '89.218.22.178:8080', '192.241.143.186:80', '113.53.29.218:38310', '36.78.131.182:39243']

def get_title(url):
    random.shuffle(proxyList)
    proxy = next(cycle(proxyList))
    print(f'proxy in use {proxy}')
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument(f'--proxy-server={proxy}')
    driver = webdriver.Chrome(options=chrome_options)
    try:
        driver.get(url)
        print(driver.title)
    except Exception:
        if proxy in proxyList:
            proxyList.remove(proxy)
        return get_title(url)

if __name__ == '__main__':
    links = ['http://www.google.com/','https://stackoverflow.com/','https://www.yahoo.com/']
    for link in links:
        get_title(link)

Answer 1

一次创建驱动程序并将其传递给函数，而不是在函数中创建驱动程序。这也意味着始终使用同一代理...

由于代理是在启动chromedriver之前作为参数传递的，因此除非创建新的驱动程序，否则无法更改。

import random
from itertools import cycle
from selenium import webdriver

proxyList = ['103.110.37.244:36022', '180.254.218.229:8080', '110.74.197.207:50632', '1.20.101.95:49001', '200.10.193.90:8080', '173.164.26.117:3128', '103.228.118.66:43002', '178.128.231.201:3128', '1.2.169.54:55312', '181.52.85.249:31487', '97.64.135.4:8080', '190.96.214.123:53251', '52.144.107.142:31923', '45.5.224.145:52035', '89.218.22.178:8080', '192.241.143.186:80', '113.53.29.218:38310', '36.78.131.182:39243']

def get_title(url, driver):
    driver.get(url)
    print(driver.title)

if __name__ == '__main__':
    random.shuffle(proxyList)
    proxy = next(cycle(proxyList))
    print(f'proxy in use {proxy}')
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument(f'--proxy-server={proxy}')
    driver = webdriver.Chrome(options=chrome_options)
    links = ['http://www.google.com/','https://stackoverflow.com/','https://www.yahoo.com/']
    for link in links:
        get_title(link, driver)

创建单个驱动程序以使用代理遍历多个站点

1 个答案: