线程缓冲区崩溃

Question

我试图通过线程和列表来实现5个元素的缓冲区。

wxNativeWindow

从理论上讲，它可以工作，但是当您超出缓冲区的限制时，通过缓冲6个值或在没有缓冲的值时尝试“ take（）”，IDE将变得无响应。我该如何解决这个问题？

Answer 1

您仅使用一个线程在缓冲区中添加元素，因此该列表包含5个项目，并且您的主线程在self.lock.wait()上无限期等待。您可以使用另一个将并行处理某些元素的线程，然后它将通知生产者线程。

例如，创建需要5个项目的使用者线程：

def consume(buffer):
    import time
    for i in range(5):
        print(threading.current_thread(), "consume", buffer.take())
        time.sleep(2)
    print(threading.current_thread(), "Bye")


buffer = Buffer1(5)
t = threading.Thread(target=consume, args=(buffer,), name="consumer")
t.start()

buffer.put(1)
buffer.put(2)
buffer.put(3)
buffer.put(4)
buffer.put(5)
buffer.put(6)
print(buffer.show_list())

Answer 2

... IDE变得无响应。我该如何解决这个问题？

您只显示了从主线程添加到缓冲区，没有任何东西可以取出。

如果缓冲区已满或变空，则下一个放置/取出将导致其条件（lock）to wait，直到有通知其继续。在您的示例中我没有看到任何信号。

缓冲区是共享资源。缓冲区和使用该缓冲区的线程需要进行良好的控制，以便每个人都可以远离其他人的方式，并拥有足够的逻辑以防止卡在某个地方。

大概您需要一个将东西放入缓冲区的线程和一个将东西从缓冲区取出的线程-两者都有足够的信号通知每个人完成对缓冲区的弄乱。

---

• 设置日志记录，以便可以通过日志消息跟踪。
• Buffer1更改：
  • 将列表更改为collections.deque以简化操作。
  • 添加了空白和完整属性
  • 添加了一个Event属性，以在进程关闭时停止放置/获取。
  • 添加了一个超时，当线程关闭
  时，等待放置/接收所有定时问题
  • 添加了有关空和满条件的通知。
• 有两个线程：一个添加到缓冲区中，另一个从缓冲区中获取。
  • 每个事件都将在未设置其事件的情况下进行添加/获取。
  • 在每次迭代中，都会获取或添加随机数量的项目。
  • 获取/放入缓冲区的条件后，完成后会通知所有侍者。
• 在主线程中：
  • 已创建一个事件-用于发信号通知线程退出
  • 创建一个Timer来限制线程执行时间-当它的 callback 函数超时时，它会设置事件并使用缓冲区的Condition（lock）来通知正在等待且空闲的任何人他们起来。
  • 创建，启动和加入线程。

---

import threading
import collections
import random
import string
import time
import logging
import sys

# logging setup
root = logging.getLogger()
root.setLevel(logging.INFO)

formatter = logging.Formatter(fmt='%(asctime)s.%(msecs)03d %(message)s',
                              datefmt='%S')

class WarningFilter(logging.Filter):
    def filter(self, record):
        return record.levelno == logging.WARNING

class InfoFilter(logging.Filter):
    def filter(self, record):
        return record.levelno == logging.INFO

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
handler.addFilter(InfoFilter())
root.addHandler(handler)

handler = logging.StreamHandler(sys.stderr)
handler.setLevel(logging.WARNING)
handler.setFormatter(formatter)
handler.addFilter(WarningFilter())
root.addHandler(handler)
# logging setup end

class Buffer1:
    '''FILO buffer.
    '''
    def __init__(self,size,evt) :
        self.content = None
        self.size = size
        self.evt = evt
        self.lock = threading.Condition()
        self.list = collections.deque()

    @property
    def full(self):
        return len(self.list) >= self.size
    @property
    def empty(self):
        return bool(self.list)
    
    def take(self) :
        with self.lock :
            while not self.empty:
                root.warning('buffer empty waiting to take')
                self.lock.wait(timeout=5)
                if self.evt.is_set():
                    help = None
                    break
            else:
                help = self.list.pop()
            self.lock.notify_all() 
        return help
        
    def put(self,v):
        success = False
        with self.lock :
            while self.full:
                root.warning('buffer full waiting to put')
                self.lock.wait(timeout=5)
                if self.evt.is_set():
                    break
            else:
                self.list.append(v)
                success = True
            self.lock.notify_all()
        return success

    def show_list(self):
        return self.list

class Prod(threading.Thread):
    '''Puts stuff onto buffer, quits on Event.

       Contrived toy - periodically puts random n items in buffer. 
    '''
    def __init__(self,buffer,evt):
        super().__init__(name='producer')
        self.buffer = buffer
        self.evt = evt
    def run(self):
        n = 0
        while not self.evt.is_set():
            howmany = random.randint(1,9)
            payload = random.sample(string.ascii_letters,howmany)
            payload = collections.deque(payload)
            root.info(f'{self.name} putting {howmany}')
            with self.buffer.lock:
                while payload and (not self.evt.is_set()):
                    c = payload.popleft()
                    root.info(f'{self.name} -----> {c}')
                    if not self.buffer.put(c):
                        root.warning(f'{self.name} last put failed')
                self.buffer.lock.notify_all()
            time.sleep(.04)
            n += 1
        root.info(f'{self.name} dying n={n}')
        with self.buffer.lock:
            self.buffer.lock.notify_all()
        root.info(f'{self.name} is done')
        
class Cons(threading.Thread):
    '''Takes stuff off of buffer, quits on Event set.

       Contrived toy - periodically takes random n items from buffer. 
    '''
    def __init__(self,buffer,evt):
        super().__init__(name='consumer')
        self.buffer = buffer
        self.evt = evt
    def run(self):
        n = 0
        while not self.evt.is_set():
            howmany = random.randint(1,9)
            root.info(f'{self.name} taking {howmany}')
            with self.buffer.lock:
                while (howmany > 0) and (not self.evt.is_set()):
                    c = self.buffer.take()
                    root.info(f'{self.name} <----- {c}')
                    howmany -= 1
                self.buffer.lock.notify_all()
            time.sleep(.04)
            n += 1
        root.info(f'{self.name} dying n={n}')
        with self.buffer.lock:
            self.buffer.lock.notify_all()
        root.info(f'{self.name} is done')

if __name__ == '__main__':

    # use an Event to shut down the whole process
    evt = threading.Event()
    buffer = Buffer1(5,evt)

    def kill(evt=evt,buffer=buffer):
        root.warning('killing everything')
        evt.set()
        with buffer.lock:
            buffer.lock.notify_all()

    # don't let this toy example run forever
    t = threading.Timer(5,kill)
    t.start()

    p1 = Prod(buffer,evt)
    c1 = Cons(buffer,evt)
    c1.start()
    p1.start()
    p1.join()
    c1.join()
    print('exit')

Answer 3

这是使用asyncio而不是线程来行使缓冲区的另一种方法。

import asyncio
import collections
import random
import string
import time
import logging
import sys



# logging setup
root = logging.getLogger()
root.setLevel(logging.INFO)

formatter = logging.Formatter(fmt='%(asctime)s.%(msecs)03d %(message)s',
                              datefmt='%S')

class WarningFilter(logging.Filter):
    def filter(self, record):
        return record.levelno == logging.WARNING

class InfoFilter(logging.Filter):
    def filter(self, record):
        return record.levelno == logging.INFO

handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
handler.setFormatter(formatter)
handler.addFilter(InfoFilter())
root.addHandler(handler)

handler = logging.StreamHandler(sys.stderr)
handler.setLevel(logging.WARNING)
handler.setFormatter(formatter)
handler.addFilter(WarningFilter())
root.addHandler(handler)

class Buffer:
    '''FILO buffer.
    '''
    def __init__(self,size,evt) :
        self.content = None
        self.size = size
        self.stop_evt = evt
        self.lock = asyncio.Condition()
        self.list = collections.deque()

    def full(self):
        return len(self.list) >= self.size
    def not_full(self):
        return len(self.list) < self.size
    def empty(self):
        return not bool(self.list)
    def not_empty(self):
        return bool(self.list)
    
    async def take(self) :
        async with self.lock:
            #root.info(f'take:lock acquired - wait for not empty')
            while self.empty():
                waiters = [thing for thing in self.lock._waiters]
                #root.warning(f'take:{waiters} waiting')
                await self.lock.wait()
                if self.stop_evt.is_set():    # shutting down
                    val = None
            else:
                #root.info('take: not empty')
                val = self.list.pop()
            self.lock.notify_all() 
        return val
        
    async def put(self,v):
        success = False
        async with self.lock:
            #root.info(f'put:lock acquired - wait for not full')
            while self.full():
                waiters = [thing for thing in self.lock._waiters]
                #root.warning(f'put:{waiters} waiting')
                await self.lock.wait()
                if self.stop_evt.is_set():    # shutting down
                    break
            else:
                #root.info('put: not full')
                self.list.append(v)
                success = True
            self.lock.notify_all()
        return success

    def show_list(self):
        return self.list
    
async def random_stuff():
    howmany = random.randint(1,9)
    payload = random.sample(string.ascii_letters,howmany)
    return collections.deque(payload)
    
async def produce(buffer,stop_evt,name):
    puts = []
    try:
        while True:
            payload = await random_stuff()
            root.warning(f'producer{name} putting {len(payload)}')
            while payload:
                c = payload.popleft()
                root.info(f'producer{name} -----> {c}')
                success = await buffer.put(c)
                if not success:
                    root.warning(f'producer{name} failed to put {c}')
                else:
                    puts.append(c)
            await asyncio.sleep(.03)
    except asyncio.CancelledError as e:
        root.warning('producer canceled')
    root.info(f'producer{name} dying n={len(puts)}')
    root.info(f'producer{name} is done')
    return puts

   
async def consume(buffer, stop_evt, name):
    '''Takes stuff off of buffer, quits on Event set.

       Contrived toy - periodically takes random n items from buffer. 
    '''
    takes = []
    try:
        while True:
            howmany = random.randint(1,9)
            msg = f'consumer{name} taking {howmany}'
            root.warning(f'{msg:>38}')
            while howmany > 0:
                c = await buffer.take()
                takes.append(c)
                msg = f'consumer{name} <----- {c}'
                root.info(f'{msg:>38}')
                howmany -= 1
            await asyncio.sleep(.02)
    except asyncio.CancelledError as e:
        root.warning('consumer canceled')
    root.info(f'consumer{name} dying n={len(takes)}')
    root.info(f'consumer{name} is done')
    return takes

async def timer(n,buffer,evt, tasks):
    root.warning('timer started')
    await asyncio.sleep(n)
    evt.set()
    root.warning('timed out - event set')
    root.warning('canceling tasks')
    for task in tasks:
        task.cancel()
        

async def main():

    loop = asyncio.get_running_loop()
    loop.set_debug(True)

    # use an Event to shut down the whole process
    evt = asyncio.Event()
    buffer = Buffer(5,evt)
    
    put_task = asyncio.create_task(produce(buffer,evt,1))
    take_task = asyncio.create_task(consume(buffer,evt,1))
    timer_task = asyncio.create_task(timer(5,buffer,evt,[put_task,take_task]))
    root.info('tasks created')
    await timer_task
    puts = await put_task
    takes = await take_task
    print('exit')
    return puts,takes,buffer.list

if __name__ == '__main__':

    puts,takes,remains = asyncio.run(main())
    puts = collections.Counter(puts)
    takes = collections.Counter(takes)
    remains = collections.Counter(remains)
    #print(remains == (puts-takes))