Question

我正在使用gevent为游泳池运行许多芹菜任务（20,000）（也是猴子修补所有）。这些任务中的每一项都会使用adwords等第三方服务来提取数据。

由于潜在的SSL错误，我一直在执行任务失败。下面是一些例外的堆栈跟踪（没有特定的顺序，这些是来自单独任务的失败）。偶尔也会遇到WantWriteError和ZeroReturnError，但EOF错误似乎最多。

使用不同的客户端库（如googleads（用于肥皂通信的suds库）以及请求和elasticsearch时会发生这些错误。我猜这些库中的一些使用urllib3，而其他库使用urllib2等。

有很多关于EOF问题和强制TLSv1的信息，但我似乎无法找到有效的解决方案。

我不确定我是否会立刻运行太多请求，如果某些事情阻塞或者是什么;任何帮助都会非常感激，我会把头发拉过来。

Traceback (most recent call last):
  ...
  File "/srv/reporting/src/reporting/stats/adwords/client.py", line 58, in _awql_report
    downloader = self._get_client(client_id).GetReportDownloader(version=self.REPORT_DOWNLOADER_VERSION)
  File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 283, in GetReportDownloader
    return ReportDownloader(self, version, server)
  File "/usr/local/lib/python2.7/dist-packages/googleads/adwords.py", line 400, in __init__
    proxy=proxy_option, cache=self._adwords_client.cache).wsdl.schema
  File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 115, in __init__
    self.wsdl = reader.open(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 150, in open
    d = self.fn(url, self.options)
  File "/usr/local/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__
    d = reader.open(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 74, in open
    d = self.download(url)
  File "/usr/local/lib/python2.7/dist-packages/suds/reader.py", line 92, in download
    fp = self.options.transport.open(Request(url))
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/https.py", line 62, in open
    return HttpTransport.open(self, request)
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 67, in open
    return self.u2open(u2request)
  File "/usr/local/lib/python2.7/dist-packages/suds/transport/http.py", line 132, in u2open
    return url.open(u2request, timeout=tm)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1216, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>



Traceback (most recent call last):
  ...
  File "/srv/reporting/src/reporting/stats/analytics/client.py", line 57, in get_access_token
    response = requests.post('https://accounts.google.com/o/oauth2/token', data)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')




Traceback (most recent call last):
  ...
    self.es.index(index=self.INDICE, doc_type=self.ROOT_CLASS.__name__, body=self.export(obj), id=obj.id)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 68, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 213, in index
    _make_path(index, doc_type, id), params=params, body=body)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 284, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py", line 44, in perform_request
    response = self.session.request(method, url, data=body, timeout=timeout or self.timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 319, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 273, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 995, in recv
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
    raise ZeroReturnError()
ZeroReturnError

Answer 1

因此，让我们通过每个回溯块来打破这个问题。第一个结束于：

  File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol>

这来自urllib2。收到EOF这一事实让我觉得服务器在你等待的时候关闭了连接＆＃34;线程＆＃34;再次从套接字读取。您可能希望使用更多time.sleep(0)来屈服于gevent。

第二次追溯来自请求：

  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
SSLError: [Errno bad handshake] (-1, 'Unexpected EOF')

[Errno bad handshake]会让我倾向于认为这是建立连接的问题，这可能是由意外的EOF引起的。这是由使用gevent引起的吗？我不确定。

最终回溯肯定来自请求，但它也来自PyOpenSSL，并且不会被urllib3或请求捕获。

  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 851, in _raise_ssl_error
    raise ZeroReturnError()
ZeroReturnError

我做了一些搜索，发现"According to the pyOpenSSL docs ZeroReturnError means that the SSL connection has been closed cleanly."这告诉我服务器再次关闭了连接，因为你花了很长时间从套接字中读取任何内容。

简而言之，我认为您需要更频繁地产生，以确保不会出现这些套接字问题。这只是一个猜测，所以带上一粒盐。

python ssl eof违反协议，wantwriteerror，zeroreturnerror发生

1 个答案: