我正在使用kubernetes python client。如果我的代码启动时kubernetes不可用,我想重试连接。
当客户端无法连接时,它会抛出看似urllib3.exceptions.MaxRetryError
的异常,所以我开始使用这样的东西:
import time
import urllib3
import kubernetes
kubernetes.config.load_kube_config()
api = kubernetes.client.CoreV1Api()
while True:
try:
w = kubernetes.watch.Watch()
for event in w.stream(api.list_pod_for_all_namespaces):
print event
except urllib3.exceptions.HTTPError:
print('retrying in 1 second')
time.sleep(1)
但那完全失败了;它的作用就像没有except
语句一样,并以:
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2743110>: Failed to establish a new connection: [Errno 111] Connection refused',))
我想也许我不了解继承以及我想的,所以我用以下内容替换上面的内容:
except urllib3.exceptions.MaxRetryError:
print('retrying in 1 second')
time.sleep(1)
以同样的方式失败。为了弄清楚发生了什么,我添加了一个全能except
并调用了pdb:
except Exception as err:
import pdb; pdb.set_trace()
从pdb
提示符,我们可以看到:
(Pdb) type(err)
<class 'urllib3.exceptions.MaxRetryError'>
......看起来很好,和mro一样:
(Pdb) import inspect
(Pdb) inspect.getmro(err.__class__)
(<class 'urllib3.exceptions.MaxRetryError'>, <class 'urllib3.exceptions.RequestError'>, <class 'urllib3.exceptions.PoolError'>, <class 'urllib3.exceptions.HTTPError'>, <type 'exceptions.Exception'>, <type 'exceptions.BaseException'>, <type 'object'>)
但尽管如此:
(Pdb) isinstance(err, urllib3.exceptions.MaxRetryError)
False
所有路径看起来都很合理:
(Pdb) urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'
(Pdb) kubernetes.client.rest.urllib3.__file__
'/usr/lib/python2.7/site-packages/urllib3/__init__.pyc'
那么......实际发生了什么?
更新
这是完整的堆栈跟踪:
Traceback (most recent call last):
File "testkube.py", line 13, in <module>
for event in w.stream(api.list_pod_for_all_namespaces):
File "/usr/lib/python2.7/site-packages/kubernetes/watch/watch.py", line 116, in stream
resp = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14368, in list_pod_for_all_namespaces
(data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
File "/usr/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 14464, in list_pod_for_all_namespaces_with_http_info
collection_formats=collection_formats)
File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 335, in call_api
_preload_content, _request_timeout)
File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 148, in __call_api
_request_timeout=_request_timeout)
File "/usr/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 371, in request
headers=headers)
File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 250, in GET
query_params=query_params)
File "/usr/lib/python2.7/site-packages/kubernetes/client/rest.py", line 223, in request
headers=headers)
File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 66, in request
**urlopen_kw)
File "/usr/lib/python2.7/site-packages/urllib3/request.py", line 87, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/usr/lib/python2.7/site-packages/urllib3/poolmanager.py", line 321, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
**response_kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
**response_kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
**response_kw)
File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.122.140', port=8443): Max retries exceeded with url: /api/v1/pods?watch=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x3d16110>: Failed to establish a new connection: [Errno 111] Connection refused',))
答案 0 :(得分:7)
您的代码与requests
包有间接依赖关系,requests
包有一个名为requests.packages
的奇怪子模块。这用于包含来自许多依赖项的复制源代码,包括urllib3
,但它们已停止这样做。他们希望保持requests.packages
以保持向后兼容性,所以现在他们做了一些奇怪的事情。
而不是requests.packages
包含urllib3
源代码的完整副本,它现在导入urllib3
并设置sys.modules['requests.packages.urllib3'] = urllib3
。根据{{1}}版本,它也可以设置许多其他requests
条目;例如,从请求2.18.4开始,源代码
sys.modules
但在2.17.0中,它确实
for package in ('urllib3', 'idna', 'chardet'):
locals()[package] = __import__(package)
# This traversal is apparently necessary such that the identities are
# preserved (requests.packages.urllib3.* is urllib3.*)
for mod in list(sys.modules):
if mod == package or mod.startswith(package + '.'):
sys.modules['requests.packages.' + mod] = sys.modules[mod]
此代码与导入的包的子模块交互不良。如果某些代码尝试执行import urllib3
sys.modules['requests.packages.urllib3'] = urllib3
import idna
sys.modules['requests.packages.idna'] = idna
import chardet
sys.modules['requests.packages.chardet'] = chardet
并且Python找不到import requests.packages.urllib3.exceptions
条目,则Python将重新创建 sys.modules['requests.packages.urllib3.exceptions']
模块并设置urllib3.exceptions
和urllib3.exceptions
到新模块(但不会触及sys.modules['requests.packages.urllib3.exceptions']
。这会生成相关类的新副本,从而导致错误。
同样原因的相关问题是5月份的reported,导致2.18.4中显示的新代码。 2.18.4不应该导致您看到的具体问题,但它仍然很脆弱,因为sys.modules['urllib3.exceptions']
的任何子模块在urllib3
螺钉加载requests.packages
时尚未加载,那些子模块会出现你今天看到的同样的问题。