队列Python线程(分段错误:11)

时间:2017-09-12 07:46:07

标签: python multithreading queue

我有这样的队列设置。我想遍历数据库查询中的所有项目并将它们传递到我的Downloader类。我的数据库连接一直在消失,最终我的程序就死了,因为我认为有太多的线程是开放的?

我收到错误:分段错误:11

有100K +项目。

如何正确解决此问题,一次处理几件事并加快处理速度?

class Downloader(threading.Thread):
    """Threaded File Downloader"""

    def __init__(self, queue, db):
        threading.Thread.__init__(self)
        self.queue = queue
        self.db = db

    def remove_unicode(self, title):
        try:
            return unicodedata.normalize('NFKD', title).encode('ascii','ignore')
        except:
            return title

    def run(self):
        while True:
            # gets the url from the queue
            row = self.queue.get()
            title = row[0]
            etc...


def main(urls):

     queue = Queue.Queue()
     # create a thread pool and give them a queue
     for i in range(5):
          t = Downloader(queue, db)
          t.setDaemon(True)
          t.start()

        # give the queue some data
     i = 1
     for url in urls:
          print i
          queue.put(url)
          i+=1
        # wait for the queue to finish
          queue.join()

if __name__ == "__main__":
     db = DatabaseUtil()
     sql = 'SELECT `Title`, `Site` from `XYZ`'
     titles = db.query(sql)
     main(titles)

0 个答案:

没有答案