Question

这就是我正在做的事情：

从文本文件中获取单词 - 每个单词都在一个单独的行中。
将http://www.和.com添加到字词以创建网址。
获取带有请求的网址。
找出它是否是免费域名（基于状态码和连接错误/其他错误）。
将免费域添加到文本文件中。
时间一切。

到目前为止，我已经开始工作，但速度非常慢。文本文件有35万字。我将如何同时或并行地执行此操作？还有哪个更适合这项任务呢？

这是我的代码：

import requests, time

start = time.time()

with open('words1.txt','r') as f:
    words = []
    for item in f:
        words.append(item.strip())

for w in words:
    url = 'http://www.'+w+'.com'
    try:
        header = {'User-Agent': 'Mozilla/5.0'}
        r = requests.get(url, headers=header)
        codes = [200,201,202,203,204,205,206,300,301,302,303,307,308,400,401,402,403,404,405,406,500,501,502,503]
        if r.status_code in codes:
            print(url,': Known Status Code > Unavailable')
        else:
            print(url,': Unknown Status Code > Probably Free')
            with open('available.txt','a') as myfile:
                myfile.write(url+'\n')
    except requests.exceptions.ConnectionError:
        print(url,' : Connection Error > Probably Free')
        with open('available.txt','a') as myfile:
            myfile.write(url+'\n')
    except requests.exceptions.HTTPError:
        print('http error')
    except requests.exceptions.Timeout:
        print('timeout error')
    except requests.exceptions.TooManyRedirects:
        print('too many redirects')

end = time.time()
print('\n')
print(end-start, 'seconds')
print((end-start)/60,'minutes')
print(((end-start)/60)/60,'hours')

谢谢！

编辑：我让它发挥作用。感谢您帮助 Kendas 和 DeepSpace ！这是一个快速测试：

100字 - 22秒

1000字 - 285秒

不是太快，但比第一次尝试快得多。

看起来像gevent + socket是要走的路。

如果您有任何关于更好/更快的建议，请告诉我。

以下是代码：

import gevent,time
from gevent import socket

start = time.time()

words = []
with open('words1000.txt','r') as f:
    for item in f:
        words.append(item.strip())

urls = ['www.{}.com'.format(w) for w in words]

jobs = [gevent.spawn(socket.gethostbyname, url) for url in urls]

gevent.joinall(jobs)

values = {url:job.value for (url,job) in zip(urls,jobs)}
freeDomains = []
for (v,job,url) in zip(values,jobs,urls):
    if job.value == None:
        freeDomains.append(url)
        with open('availableds.txt','a') as myFile:
            myFile.write(url+'\n')

print(freeDomains)
end = time.time()
print(end-start,'seconds')
print((end-start)/60,'minutes')
print((end-start)/3600,'hours')

Answer 1

grequests（并发版本的请求）使这很容易。它还有助于使用.format而不是每次迭代重新定义header

import grequests

def exception_handler(request, exception):
    print(exception)

with open('words1.txt','r') as f:
    words = []
    for item in f:
        words.append(item.strip())
urls = ['http://www.{}.com'.format(w) for w in words]

header = {'User-Agent': 'Mozilla/5.0'}

requests = [grequests.get(url) for url in urls]

responses = grequests.map(requests, exception_handler=exception_handler)

for resp in responses:
    if resp:
         print(resp.status_code)

如何同时或并行地在Python中执行此操作（检查请求状态代码）？

1 个答案: