Question

我使用代理使用urllib2发送了数千个请求。我在执行时收到了以下许多错误：

urlopen error [Errno 99] Cannot assign requested address

我读到here可能是因为套接字已被绑定。是这样的吗？有关如何解决此问题的任何建议吗？

Answer 1

以下是我之前准备的一个类似问题的答案......早些时候...... Socket in use error when reusing sockets

错误是不同的，但基本问题可能是相同的：您正在消耗所有可用端口并尝试在TIME_WAIT状态结束之前重用它们。

[编辑：回应评论]

如果它在您的应用程序的功能/规范范围内，一个明显的策略是控制连接速率以避免这种情况。

或者，您可以使用httplib模块。 httplib.HTTPConnection()允许您指定source_address元组，您可以使用该元组指定进行连接的端口，例如这将从localhost：9999：

连接到localhost：1234

import httplib
conn = httplib.HTTPConnection('localhost:1234', source_address=('localhost',9999))
conn.request('GET', '/index.html')

然后就像我之前的回答中所描述的那样管理源端口分配。如果您使用的是Windows，则可以使用此方法绕过端口1024-5000的默认范围。

（当然）有一个上限是你能够建立多少个连接，并且有什么样的应用程序需要快速连续地建立成千上万的连接是值得怀疑的。

Answer 2

正如 mhawke 所建议的那样，TIME_WAIT的问题似乎最有可能发生。针对您的情况的系统范围修复可以是调整内核参数，以便更频繁地清理这些连接。两个选项：

$ sysctl net.ipv4.tcp_tw_recycle=1

这将让内核重用TIME_WAIT状态的连接。这可能会导致NAT设置出现问题。另一个是：

$ sysctl net.ipv4.tcp_max_orphans=8192
$ sysctl net.ipv4.tcp_orphan_retries=1

这告诉内核保留最多8192个连接没有连接到任何用户进程的连接，并且只能在杀死TCP连接之前重试一次。

请注意，这些不是永久性更改。将设置添加到/etc/sysctl.conf以使其永久化。

http://code.google.com/p/lusca-cache/issues/detail?id=89#c4
http://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.kernel.obscure.html

Answer 3

我遇到了类似的问题，但是使用了python的请求库虽然使用了POST命令!! 更糟糕的是，我在每个执行程序上使用多处理来发布到服务器。因此，在几秒钟内创建了数千个连接，每个连接都需要几秒钟才能从TIME_WAIT更改状态并释放下一组连接的端口。

在互联网上提供的禁用保持活动的所有可用解决方案中，与request.Session（）等一起使用，我发现this回答是有效的，它使用了＆＃39;连接＆＃39; ：＆＃39;关闭＆＃39;配置为头参数。您可能需要将标题内容放在post命令之外的separte行中。

import numpy as np
import warnings

def interp_along_axis(y, x, newx, axis, inverse=False, method='linear'):
    """ Interpolate vertical profiles, e.g. of atmospheric variables
    using vectorized numpy operations

    This function assumes that the x-xoordinate increases monotonically

    ps:
    * Updated to work with irregularly spaced x-coordinate.
    * Updated to work with irregularly spaced newx-coordinate
    * Updated to easily inverse the direction of the x-coordinate
    * Updated to fill with nans outside extrapolation range
    * Updated to include a linear interpolation method as well
        (it was initially written for a cubic function)

    Peter Kalverla
    March 2018

    --------------------
    More info:
    Algorithm from: http://www.paulinternet.nl/?page=bicubic
    It approximates y = f(x) = ax^3 + bx^2 + cx + d
    where y may be an ndarray input vector
    Returns f(newx)

    The algorithm uses the derivative f'(x) = 3ax^2 + 2bx + c
    and uses the fact that:
    f(0) = d
    f(1) = a + b + c + d
    f'(0) = c
    f'(1) = 3a + 2b + c

    Rewriting this yields expressions for a, b, c, d:
    a = 2f(0) - 2f(1) + f'(0) + f'(1)
    b = -3f(0) + 3f(1) - 2f'(0) - f'(1)
    c = f'(0)
    d = f(0)

    These can be evaluated at two neighbouring points in x and
    as such constitute the piecewise cubic interpolator.
    """

    # View of x and y with axis as first dimension
    if inverse:
        _x = np.moveaxis(x, axis, 0)[::-1, ...]
        _y = np.moveaxis(y, axis, 0)[::-1, ...]
        _newx = np.moveaxis(newx, axis, 0)[::-1, ...]
    else:
        _y = np.moveaxis(y, axis, 0)
        _x = np.moveaxis(x, axis, 0)
        _newx = np.moveaxis(newx, axis, 0)

    # Sanity checks
    if np.any(_newx[0] < _x[0]) or np.any(_newx[-1] > _x[-1]):
        # raise ValueError('This function cannot extrapolate')
        warnings.warn("Some values are outside the interpolation range. "
                      "These will be filled with NaN")
    if np.any(np.diff(_x, axis=0) < 0):
        raise ValueError('x should increase monotonically')
    if np.any(np.diff(_newx, axis=0) < 0):
        raise ValueError('newx should increase monotonically')

    # Cubic interpolation needs the gradient of y in addition to its values
    if method == 'cubic':
        # For now, simply use a numpy function to get the derivatives
        # This produces the largest memory overhead of the function and
        # could alternatively be done in passing.
        ydx = np.gradient(_y, axis=0, edge_order=2)

    # This will later be concatenated with a dynamic '0th' index
    ind = [i for i in np.indices(_y.shape[1:])]

    # Allocate the output array
    original_dims = _y.shape
    newdims = list(original_dims)
    newdims[0] = len(_newx)
    newy = np.zeros(newdims)

    # set initial bounds
    i_lower = np.zeros(_x.shape[1:], dtype=int)
    i_upper = np.ones(_x.shape[1:], dtype=int)
    x_lower = _x[0, ...]
    x_upper = _x[1, ...]

    for i, xi in enumerate(_newx):
        # Start at the 'bottom' of the array and work upwards
        # This only works if x and newx increase monotonically

        # Update bounds where necessary and possible
        needs_update = (xi > x_upper) & (i_upper+1<len(_x))
        # print x_upper.max(), np.any(needs_update)
        while np.any(needs_update):
            i_lower = np.where(needs_update, i_lower+1, i_lower)
            i_upper = i_lower + 1
            x_lower = _x[[i_lower]+ind]
            x_upper = _x[[i_upper]+ind]

            # Check again
            needs_update = (xi > x_upper) & (i_upper+1<len(_x))

        # Express the position of xi relative to its neighbours
        xj = (xi-x_lower)/(x_upper - x_lower)

        # Determine where there is a valid interpolation range
        within_bounds = (_x[0, ...] < xi) & (xi < _x[-1, ...])

        if method == 'linear':
            f0, f1 = _y[[i_lower]+ind], _y[[i_upper]+ind]
            a = f1 - f0
            b = f0

            newy[i, ...] = np.where(within_bounds, a*xj+b, np.nan)

        elif method=='cubic':
            f0, f1 = _y[[i_lower]+ind], _y[[i_upper]+ind]
            df0, df1 = ydx[[i_lower]+ind], ydx[[i_upper]+ind]

            a = 2*f0 - 2*f1 + df0 + df1
            b = -3*f0 + 3*f1 - 2*df0 - df1
            c = df0
            d = f0

            newy[i, ...] = np.where(within_bounds, a*xj**3 + b*xj**2 + c*xj + d, np.nan)

        else:
            raise ValueError("invalid interpolation method"
                             "(choose 'linear' or 'cubic')")

    if inverse:
        newy = newy[::-1, ...]

    return np.moveaxis(newy, 0, axis)

试试请求库。

Python urllib2：无法分配请求的地址

3 个答案: