Question

我在Python 2.7.13上运行线程脚本时遇到了奇怪的问题。有时整个python.exe只是崩溃而没有错误消息，有时脚本只是挂起并停止运行，但有时我实际上得到了错误消息。

    Exception in thread Thread-370:
    Traceback (most recent call last):
      File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
        self.run()
      File "C:\Python27\lib\threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File ".\1024.py", line 38, in loadData
        result = play_scraper.similar(app_id, results=60)
      File "C:\Python27\lib\site-packages\play_scraper\api.py", line 92, in similar
        return s.similar(app_id, **kwargs)
      File "C:\Python27\lib\site-packages\play_scraper\scraper.py", line 435, in similar
        response = send_request('GET', url)
      File "C:\Python27\lib\site-packages\play_scraper\utils.py", line 128, in send_request
        verify=verify)
      File "C:\Python27\lib\site-packages\requests\sessions.py", line 501, in get
        return self.request('GET', url, **kwargs)
      File "C:\Python27\lib\site-packages\requests\sessions.py", line 488, in request
        resp = self.send(prep, **send_kwargs)
      File "C:\Python27\lib\site-packages\requests\sessions.py", line 609, in send
        r = adapter.send(request, **kwargs)
      File "C:\Python27\lib\site-packages\requests\adapters.py", line 423, in send
        timeout=timeout
      File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 594, in urlopen
        chunked=chunked)
      File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 350, in _make_request
        self._validate_conn(conn)
      File "C:\Python27\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 835, in _validate_conn
        conn.connect()
      File "C:\Python27\lib\site-packages\requests\packages\urllib3\connection.py", line 281, in connect
        conn = self._new_conn()
      File "C:\Python27\lib\site-packages\requests\packages\urllib3\connection.py", line 138, in _new_conn
        (self.host, self.port), self.timeout, **extra_kw)
      File "C:\Python27\lib\site-packages\requests\packages\urllib3\util\connection.py", line 79, in create_connection
        sock = socket.socket(af, socktype, proto)
      File "C:\Python27\lib\site-packages\gevent\_socket2.py", line 124, in __init__
        self._read_event = io(fileno, 1)
      File "gevent.libev.corecext.pyx", line 487, in gevent.libev.corecext.loop.io (src/gevent/libev/gevent.corecext.c:6680)
      File "gevent.libev.corecext.pyx", line 835, in gevent.libev.corecext.io.__init__ (src/gevent/libev/gevent.corecext.c:11088)
    IOError: cannot watch more than 1024 sockets

我的脚本看起来像这样

import requests
from threading import Thread
import play_scraper

with open('apps.txt') as f:
    app_idList = f.read().splitlines()

checkedIds = 0

def safe_print(content):
    print "{0}\n".format(content),

def loadData (threadName,app_id):

    global checkedIds

    safe_print(threadName + str(checkedIds) + " Checking similar apps to " + app_id)
    result = play_scraper.similar(app_id, results=60)

    checkedIds += 1

for app_id in app_idList:

    t = Thread(target=loadData, args=("Thread #0: ",app_id))
    t.start()
    t.join()

在大约365-375个循环之后，我收到上面的错误消息。我在我的项目中使用了play_scraper模块，违规代码就像这样

def send_request(method, url, data=None, params=None, headers=None, verify=True):

    data = {} if data is None else data
    params = {} if params is None else params
    headers = default_headers() if headers is None else headers
    if not data and method == 'POST':
        data = generate_post_data()

    try:
        response = requests.request(
            method=method,
            url=url,
            data=data,
            params=params,
            headers=headers,
            verify=verify)
        if not response.status_code == requests.codes.ok:
            response.raise_for_status()
    except requests.exceptions.RequestException as e:
        log.error(e)
        raise

    return response

我在某处读到这个问题可能是由于打开连接套接字而使用会话会解决问题。我将该功能编辑为以下内容，但我仍然遇到同样的问题。

def send_request(method, url, data=None, params=None, headers=None, verify=True):

    data = {} if data is None else data
    params = {} if params is None else params
    headers = default_headers() if headers is None else headers
    if not data and method == 'POST':
        data = generate_post_data()

    try:
        s = requests.Session()
        if method == 'POST':
            response = s.post(
                url=url,
                data=data,
                params=params,
                headers=headers,
                verify=verify)
        else:
             response = s.get(
                url=url,
                data=data,
                params=params,
                headers=headers,
                verify=verify)
        if not response.status_code == requests.codes.ok:
            response.raise_for_status()
    except requests.exceptions.RequestException as e:
        log.error(e)
        raise
    finally:
        s.close()

    return response

如果我运行一个简单的循环脚本，一切运行正常。

    import play_scraper


    with open('apps.txt') as f:
        app_idList = f.read().splitlines()

    checkedIds = 0

    for app_id in app_idList:
        print str(checkedIds ) + " Checking similar apps to " + app_id
        result = play_scraper.similar(app_id, results=60)
        checkedIds += 1

可以在此处找到游戏刮刀模块 https://github.com/danieliu/play-scraper

我该如何解决此问题？

＆＃34; IOError：无法观看超过1024个套接字＆＃34;使用线程时

0 个答案: