使用urllib3.PoolManager时看到ClosedPoolError

时间:2017-07-28 08:58:18

标签: python multithreading urllib3

当使用urllib3.PoolManager(python 2.7,urllib3版本:1.21)时,我经常遇到ClosedPoolError异常。在阅读完代码之后,我倾向于认为这是预期的,因为返回的连接对象在使用时可能已被池的缓存逐出。

我已将错误复制到以下脚本中:

https://pastebin.com/aJggVHuv

#!/bin/bash

set -ex

TEST_DIR=/tmp/urllib3_bug
mkdir -p $TEST_DIR
cd $TEST_DIR
virtualenv .virtualenv
. .virtualenv/bin/activate
pip install urllib3==1.21

# Starts 3 threads with pool size = 2
echo 'from urllib3 import PoolManager
import threading
from threading import Thread

pool_manager = PoolManager(num_pools=2)


def start(url):
    print "Getting", url, "on", threading.current_thread().ident
    return pool_manager.urlopen("GET", url)


thread1 = Thread(target=start, args=["http://localhost:9998"])
thread2 = Thread(target=start, args=["http://localhost:9999"])
thread3 = Thread(target=start, args=["http://localhost:10000"])

thread1.start()
import time
time.sleep(1)
thread2.start()
thread3.start()
' > start.py

# Simulates preemption by sleeping the thread
echo '303a304,308
>         import time
>         import threading
>         print "Simulating preemption on", threading.current_thread().ident
>         time.sleep(3)
>         print "Waking up on", threading.current_thread().ident' | patch $TEST_DIR/.virtualenv/local/lib/python2.7/site-packages/urllib3/poolmanager.py -
rm -f $TEST_DIR/.virtualenv/local/lib/python2.7/site-packages/urllib3/poolmanager.pyc


python start.py

我创建一个大小为2的池,然后运行3个线程,每个线程连接到不同的URL。第一个线程应该遇到ClosedPoolError。

(补丁插入sleep()以在第一个线程上可靠地重现抢占。)

0 个答案:

没有答案