如何在python-requests lib中指定“socket_options”,因为urllib3 v1.8.3已经添加了“socket_options”功能?

时间:2014-07-04 07:59:51

标签: python-requests urllib3

urllib3 V1.8.3添加了“socket_options”功能。对我来说,我想在请求lib中指定(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)。但我不知道怎么做。 Here是urllib3的版本更改日志。 以下是urllib3中新功能的示例:

    def test_socket_options(self):
    """Test that connections accept socket options."""
    # This test needs to be here in order to be run. socket.create_connection actually tries to
    # connect to the host provided so we need a dummyserver to be running.
    pool = HTTPConnectionPool(self.host, self.port, socket_options=[
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
    ])
    s = pool._new_conn()._new_conn()  # Get the socket
    using_keepalive = s.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE) > 0
    self.assertTrue(using_keepalive)
    s.close()

如果将1.8.3版本合并到新请求lib中,如何指定参数。 任何答案将不胜感激。

这是我的代码:

import requests
import socket

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager

class SourceAddressAdapter(HTTPAdapter):
    def __init__(self, source_address, **kwargs):
        self.source_address = source_address
        super(SourceAddressAdapter, self).__init__(**kwargs)
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       socket_options=[(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)],
                                       source_address=self.source_address,
                                       )
http_session = requests.Session()
http_session.mount('http://', SourceAddressAdapter(('0.0.0.0', 1238)))
print http_session.get('http://10.0.10.7')

这是错误:

    Traceback (most recent call last):
  File "./tt.py", line 27, in <module>
    print http_session.get('http://10.0.10.7')
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/adapters.py", line 375, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.0.10.7', port=80): Max retries exceeded with url: / (Caused by <class 'socket.error'>: [Errno 98] Address already in use)

=============================================== ================================

我曾尝试使用urllib3 v1.9设置(socket.SOL_SOCKET,socket.SO_REUSEADDR,1),但失败了。无法重用time_wait状态的源端口的原因可能是“(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)”应在“socket.bind”之前配置。因此,即使我在“HTTPConnectionPool”中设置了REUSEADDR参数,也无法使用time_wait状态的src端口。太伤心了..

这是我的尝试:

#!/usr/bin/python
import socket
from urllib3 import (
    encode_multipart_formdata,
    HTTPConnectionPool,
    )

pool = HTTPConnectionPool('10.0.10.7', 80, 
                          socket_options=[(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)],
                          source_address=('10.0.10.11',1239))
r=pool.request('GET','/')

这是错误:

    Traceback (most recent call last):
  File "./aa.py", line 10, in <module>
    r=pool.request('GET','/')
  File "build/bdist.linux-x86_64/egg/urllib3/request.py", line 68, in request
  File "build/bdist.linux-x86_64/egg/urllib3/request.py", line 81, in request_encode_url
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 579, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 579, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 579, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/connectionpool.py", line 559, in urlopen
  File "build/bdist.linux-x86_64/egg/urllib3/util/retry.py", line 265, in increment
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.0.10.7', port=80): Max retries exceeded with url: / (Caused by ProtocolError('Connection aborted.', error(98, 'Address already in use')))

2 个答案:

答案 0 :(得分:1)

假设您已正确地将urllib3合并到请求中并且没有任何中断,您将使用传输适配器:

from requests.adapters import HTTPAdapter

class SockOpsAdapter(HTTPAdapter):
    def __init__(self, options, **kwargs):
        self.options = options

        super(SourceAddressAdapter, self).__init__(**kwargs)

    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       socket_options=self.options)

然后你就这样使用它:

import requests
from requests.packages.urllib3.connection import HTTPConnection

options = HTTPConnection.default_socket_options + [
              (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1),
          ]

s = requests.Session()
s.mount('http://', SockOpsAdapter(options))
s.mount('https://', SockOpsAdapter(options))

答案 1 :(得分:1)

我无法得到@Lukasa的工作答案,但我对其进行了修改并让以下工作,

import requests
import socket
from requests.adapters import HTTPAdapter
from requests.adapters import PoolManager
from requests.packages.urllib3.connection import HTTPConnection

class SockOpsAdapter(HTTPAdapter):
  def __init__(self, options, **kwargs):
    self.options = options
    super(SockOpsAdapter, self).__init__()
  def init_poolmanager(self, connections, maxsize, block=False):
    print "init_poolmanager"
    self.poolmanager = PoolManager(num_pools=connections,
                                   maxsize=maxsize,
                                   block=block,
                                   socket_options=self.options)

options =  HTTPConnection.default_socket_options + [
              (socket.SOL_SOCKET, socket.SO_REUSEADDR, 1),
          ]

print "build session"
s = requests.Session()
s.mount('http://', SockOpsAdapter(options))
s.mount('https://', SockOpsAdapter(options))

for i in xrange(0, 10):
  print "sending request %i" % i
  url = 'http://host:port' #put in a host/port here
  headers = {'Content-Type':'text/plain', 'Accept':'text/plain'}
  post_status = s.get(url, headers=headers)
  print "Post Status Code = %s" % str(post_status.status_code)
  print post_status.content[0:50]