Question

我试图利用雅虎财经的多线程技术将库存数据存储到SQL中。但是，我收到以下错误：

*Exception in thread Thread-3091:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "todatabase.py", line 19, in th
    htmltext = urllib.urlopen(base).read()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 87, in urlopen
    return opener.open(url)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 213, in open
    return getattr(self, name)(url)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 350, in open_http
    h.endheaders(data)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 893, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 855, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 832, in connect
    self.timeout, self.source_address)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 557, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
IOError: [Errno socket error] [Errno 8] nodename nor servname provided, or not known*

这是我的代码：

from threading import Thread
import sqlite3
import urllib
import re

conn = sqlite3.connect('stock.sqlite')
cur = conn.cursor()

cur.execute('''CREATE TABLE IF NOT EXISTS Stock
    (symbol TEXT UNIQUE PRIMARY KEY, price NUMERIC) ''')

dic = {}

def th(ur):
    base = "http://finance.yahoo.com/q?s=" + ur
    regex = '<span id="yfs_l84_[^.]*">(.+?)</span>'
    pattern = re.compile(regex)
    htmltext = urllib.urlopen(base).read()
    results = re.findall(pattern, htmltext)

    try:
        dic[ur] = results[0]
    except:
        print 'got a error!'

symbolslist = open("symbols.txt").read()
symbolslist = symbolslist.split("\n")
threadlist = []

for u in symbolslist:
    t = Thread(target = th, args = (u,))
    t.start()
    threadlist.append(t)

for b in threadlist:
    b.join()

for key, value in dic.items():
    print key, value

    cur.execute('INSERT INTO Stock(symbol,price) VALUES (?,?)',(key,value))
    conn.commit()

cur.close()

我认为这个错误可能出在多线程部分，因为我可以在不使用多线程的情况下获得数据，但速度很慢。

对于多线程和这个错误，我最终只得到200+（符号，价格）而不是3145.

我尝试更改DNS和IP，但没有解决它。

Answer 1

我记得我曾遇到过多线程和大量插座打开的问题。另外一把锁解决了我的问题。但是，我并没有试图找到实际问题。 urllib doc没有提及线程安全性。你可以尝试这样的事情：

global_lock = threading.Lock()
...
def th(ur):
    ...
    with global_lock:
        fd = urllib.urlopen(base)
    with fd:
        htmltext = fd.read()

修改

您也可以使用像（例如）龙卷风或asyncio这样的库来使用单线程（async IO）代码。

顺便说一句，通过在每个线程中使用sqlite连接，您可以在相应的线程中检索它后立即存储已删除的数据。

Answer 2

我也遇到了这个错误。我只是为每个线程添加一些休眠时间，问题得到解决。我用time.sleep（0.1）。

IOError：[Errno套接字错误] [Errno 8]提供nodename或servname，或者未知

2 个答案: