Question

有没有办法在Python中为多线程目的实现锁定，acquire方法可以有任意超时？到目前为止我找到的唯一可行的解决方案是使用轮询，

我觉得不雅和低效
不保留锁定的有限等待/进度保证作为临界区问题的解决方案

有没有更好的方法来实现这个？

Answer 1

详细说明史蒂文的评论建议：

import threading
import time

lock = threading.Lock()
cond = threading.Condition(threading.Lock())

def waitLock(timeout):
    with cond:
        current_time = start_time = time.time()
        while current_time < start_time + timeout:
            if lock.acquire(False):
                return True
            else:
                cond.wait(timeout - current_time + start_time)
                current_time = time.time()
    return False

需要注意的事项：

有两个threading.Lock()个对象，一个是threading.Condition()的内部对象。
操纵cond时，获取锁定;但是，wait()操作会将其解锁，因此任意数量的线程都可以观看它。
等待嵌入在跟踪时间的for循环中。 threading.Condition可能会因超时之外的原因而收到通知，因此您仍需要跟踪时间，如果您确实希望它过期。
即使有条件，你仍然'轮询'真正的锁，因为它可能有多个线程被唤醒并争夺锁。如果lock.acquire失败，则循环返回等待。
此waitLock函数的调用者应该跟lock.release() cond.notify()一起使用，以便等待它的其他线程通知他们应该重试获取锁。这未在示例中显示。

Answer 2

我的版本使用线程安全队列http://docs.python.org/2/library/queue.html及其支持超时的put / get方法。

到目前为止工作正常，但如果有人可以对其进行同行评审，我将不胜感激。

"""
Thread-safe lock mechanism with timeout support module.
"""

from threading import ThreadError, current_thread
from Queue import Queue, Full, Empty


class TimeoutLock(object):
    """
    Thread-safe lock mechanism with timeout support.
    """

    def __init__(self, mutex=True):
        """
        Constructor.
        Mutex parameter specifies if the lock should behave like a Mutex, and
        thus use the concept of thread ownership.
        """
        self._queue = Queue(maxsize=1)
        self._owner = None
        self._mutex = mutex

    def acquire(self, timeout=0):
        """
        Acquire the lock.
        Returns True if the lock was succesfully acquired, False otherwise.

        Timeout:
        - < 0 : Wait forever.
        -   0 : No wait.
        - > 0 : Wait x seconds.
        """
        th = current_thread()
        try:
            self._queue.put(
                th, block=(timeout != 0),
                timeout=(None if timeout < 0 else timeout)
            )
        except Full:
            return False

        self._owner = th
        return True

    def release(self):
        """
        Release the lock.
        If the lock is configured as a Mutex, only the owner thread can release
        the lock. If another thread attempts to release the lock a
        ThreadException is raised.
        """
        th = current_thread()
        if self._mutex and th != self._owner:
            raise ThreadError('This lock isn\'t owned by this thread.')

        self._owner = None
        try:
            self._queue.get(False)
            return True
        except Empty:
            raise ThreadError('This lock was released already.')

Answer 3

我怀疑这可以做到。

如果你想在没有任何轮询的情况下实现它，那么你需要操作系统知道线程被阻塞，操作系统需要知道超时，以便在一段时间后解锁线程。为此，操作系统中已经存在支持;你无法在Python级别实现这一点。

（你可以让线程在操作系统级别或应用程序级别被阻止，并且有一种机制可以在适当的时候被不同的线程唤醒，但是你需要其他线程才能有效地进行轮询）

一般来说，无论如何你都没有真正有效的等待/进度保证，因为你的线程必须等待无限时间才能进行上下文切换，以便它注意到它已被解锁。因此，除非您可以对CPU争用的数量设置上限，否则您将无法使用超时来达到任何硬实时截止日期。但是你可能不需要它，否则你不会梦想使用Python中实现的锁。

由于Python GIL（全局解释器锁定），那些基于轮询的解决方案可能不像你想象的那样效率低下或严重无关（取决于它们是如何实现的）（假设你使用的是CPython）或PyPy）。

一次只有一个线程运行，根据定义，你想要运行另一个线程（持有你正在等待的锁的线程）。 GIL由一个线程保持一段时间来执行一堆字节码，然后丢弃并重新获取以给其他人一个机会。因此，如果被阻塞的超时线程只是在一个循环中检查时间并屈服于其他线程，它只会在它获得GIL时经常唤醒，然后几乎立即将其丢弃给其他人并阻塞再次GIL。因为这个线程无论如何只能在GIL转弯时才会被唤醒，它也会在超时到期后立即进行检查，因为即使超时是神奇的完美，它也能够恢复执行。

这会导致很多低效率的唯一一次是你的线程被阻塞等待锁定线程，这个线程被阻塞等待一些不能由另一个Python线程引起的事情（比如在IO上阻塞），并没有其他可运行的Python线程。然后你的轮询超时真的会在那里反复检查时间，如果你预计这种情况会发生很长一段时间，这可能会很糟糕。

Answer 4

如果有人需要Python> = 3.2 API：

import threading
import time


class Lock(object):
    _lock_class = threading.Lock

    def __init__(self):
        self._lock = self._lock_class()
        self._cond = threading.Condition(threading.Lock())

    def acquire(self, blocking=True, timeout=-1):
        if not blocking or timeout == 0:
            return self._lock.acquire(False)
        cond = self._cond
        lock = self._lock
        if timeout < 0:
            with cond:
                while True:
                    if lock.acquire(False):
                        return True
                    else:
                        cond.wait()
        else:
            with cond:
                current_time = time.time()
                stop_time = current_time + timeout
                while current_time < stop_time:
                    if lock.acquire(False):
                        return True
                    else:
                        cond.wait(stop_time - current_time)
                        current_time = time.time()
                return False

    def release(self):
        with self._cond:
            self._lock.release()
            self._cond.notify()

    __enter__ = acquire

    def __exit__(self, t, v, tb):
        self.release()


class RLock(Lock):
    _lock_class = threading.RLock

Answer 5

我接受了SingleNegationElimination的回答并创建了一个类，可以通过以下方式在with语句中使用：

global_lock = timeout_lock()
...

with timeout_lock(owner='task_name', lock=global_lock):
    do()
    some.stuff()

这样，如果超时超时（默认值= 1秒），它只会警告并显示锁的所有者以进行调查。

以这种方式使用它并在超时后抛出异常：

with timeout_lock(owner='task_name', lock=global_lock, raise_on_timeout=True):
    do()
    some.stuff()

timeout_lock.lock()实例必须创建一次，并且可以跨线程使用。

这是课程 - 它对我有用，但随意评论和改进：

class timeout_lock:
    ''' taken from https://stackoverflow.com/a/8393033/1668622
    '''
    class lock:
        def __init__(self):
            self.owner = None
            self.lock = threading.Lock()
            self.cond = threading.Condition()

        def _release(self):
            self.owner = None
            self.lock.release()
            with self.cond:
                self.cond.notify()

    def __init__(self, owner, lock, timeout=1, raise_on_timeout=False):
        self._owner = owner
        self._lock = lock
        self._timeout = timeout
        self._raise_on_timeout = raise_on_timeout

    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, type, value, tb):
        ''' will only be called if __enter__ did not raise '''
        self.release()

    def acquire(self):
        if self._raise_on_timeout:
            if not self._waitLock():
                raise RuntimeError('"%s" could not aquire lock within %d sec'
                                   % (self._owner, self._timeout))
        else:
            while True:
                if self._waitLock():
                    break
                print('"%s" is waiting for "%s" and is getting bored...'
                      % (self._owner, self._lock.owner))
        self._lock.owner = self._owner

    def release(self):
        self._lock._release()

    def _waitLock(self):
        with self._lock.cond:
            _current_t = _start_t = time.time()
            while _current_t < _start_t + self._timeout:
                if self._lock.lock.acquire(False):
                    return True
                else:
                    self._lock.cond.wait(self._timeout - _current_t + _start_t)
                    _current_t = time.time()
        return False

为了确保线程真的不会干扰并且不等待尽快通知我写了一个小的多线程测试，它将总结运行所有线程所需的时间：

def test_lock_guard():
    import random

    def locking_thread_fn(name, lock, duration, timeout):
        with timeout_lock(name, lock, timeout=timeout):
            print('%x: "%s" begins to work..' % (threading.get_ident(), name))
            time.sleep(duration)
            print('%x: "%s" finished' % (threading.get_ident(), name))

    _lock = timeout_lock.lock()

    _threads = []
    _total_d = 0
    for i in range(3):
        _d = random.random() * 3
        _to = random.random() * 2
        _threads.append(threading.Thread(
            target=locking_thread_fn, args=('thread%d' % i, _lock, _d, _to)))
        _total_d += _d

    _t = time.time()

    for t in _threads: t.start()
    for t in _threads: t.join()

    _t = time.time() - _t

    print('duration: %.2f sec / expected: %.2f (%.1f%%)'
          % (_t, _total_d, 100 / _total_d * _t))

输出是：

7f940fc2d700: "thread0" begins to work..
"thread2" is waiting for "thread0" and is getting bored...
"thread2" is waiting for "thread0" and is getting bored...
"thread2" is waiting for "thread0" and is getting bored...
7f940fc2d700: "thread0" finished
7f940f42c700: "thread1" begins to work..
"thread2" is waiting for "thread1" and is getting bored...
"thread2" is waiting for "thread1" and is getting bored...
7f940f42c700: "thread1" finished
"thread2" is waiting for "None" and is getting bored...
7f940ec2b700: "thread2" begins to work..
7f940ec2b700: "thread2" finished
duration: 5.20 sec / expected: 5.20 (100.1%)

Answer 6

好的，这已经在 python 3.2 或更高版本中实现： https://docs.python.org/3/library/threading.html 寻找线程。 TIMEOUT_MAX

但是我对测试用例的版本进行了改进……虽然如果您使用的是py3.2或更高版本，这已经很浪费时间：

from unittest.mock import patch, Mock
import unittest

import os
import sys
import logging
import traceback
import threading
import time

from Util import ThreadingUtil

class ThreadingUtilTests(unittest.TestCase):

    def setUp(self):
        pass

    def tearDown(self):
        pass

    # https://www.pythoncentral.io/pythons-time-sleep-pause-wait-sleep-stop-your-code/
    def testTimeoutLock(self):

        faulted = [False, False, False]

        def locking_thread_fn(threadId, lock, duration, timeout):
            try:
                threadName = "Thread#" + str(threadId)
                with ThreadingUtil.TimeoutLock(threadName, lock, timeout=timeout, raise_on_timeout=True):
                    print('%x: "%s" begins to work..' % (threading.get_ident(), threadName))
                    time.sleep(duration)
                    print('%x: "%s" finished' % (threading.get_ident(), threadName))
            except:
                faulted[threadId] = True

        _lock = ThreadingUtil.TimeoutLock.lock()

        _sleepDuration = [5, 10, 1]
        _threads = []

        for i in range(3):
            _duration = _sleepDuration[i]
            _timeout = 6
            print("Wait duration (sec): " + str(_duration) + ", Timeout (sec): " + str(_timeout))
            _worker = threading.Thread(
                                        target=locking_thread_fn, 
                                        args=(i, _lock, _duration, _timeout)
                                    )
            _threads.append(_worker)

        for t in _threads: t.start()
        for t in _threads: t.join()

        self.assertEqual(faulted[0], False)
        self.assertEqual(faulted[1], False)
        self.assertEqual(faulted[2], True)

现在在“ Util”文件夹下，我有“ ThreadingUtil.py”：

import time
import threading

# https://stackoverflow.com/questions/8392640/how-to-implement-a-lock-with-a-timeout-in-python-2-7
# https://docs.python.org/3.4/library/asyncio-sync.html#asyncio.Condition
# https://stackoverflow.com/questions/28664720/how-to-create-global-lock-semaphore-with-multiprocessing-pool-in-python
# https://hackernoon.com/synchronization-primitives-in-python-564f89fee732

class TimeoutLock(object):
    ''' taken from https://stackoverflow.com/a/8393033/1668622
    '''
    class lock:
        def __init__(self):
            self.owner = None
            self.lock = threading.Lock()
            self.cond = threading.Condition()

        def _release(self):
            self.owner = None
            self.lock.release()
            with self.cond:
                self.cond.notify()

    def __init__(self, owner, lock, timeout=1, raise_on_timeout=False):
        self._owner = owner
        self._lock = lock
        self._timeout = timeout
        self._raise_on_timeout = raise_on_timeout

    # http://effbot.org/zone/python-with-statement.htm
    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, type, value, tb):
        ''' will only be called if __enter__ did not raise '''
        self.release()

    def acquire(self):
        if self._raise_on_timeout:
            if not self._waitLock():
                raise RuntimeError('"%s" could not aquire lock within %d sec'
                                   % (self._owner, self._timeout))
        else:
            while True:
                if self._waitLock():
                    break
                print('"%s" is waiting for "%s" and is getting bored...'
                      % (self._owner, self._lock.owner))
        self._lock.owner = self._owner

    def release(self):
        self._lock._release()

    def _waitLock(self):
        with self._lock.cond:
            _current_t = _start_t = time.time()
            while _current_t < _start_t + self._timeout:
                if self._lock.lock.acquire(False):
                    return True
                else:
                    self._lock.cond.wait(self._timeout - _current_t + _start_t)
                    _current_t = time.time()
        return False

Answer 7

基于已经接受的答案和this idea for context hybrid manager/decorators，我实现了一个同时具有上下文管理器和装饰器接口的超时锁定（在Python 2.7中有效）。此外，当用作上下文管理器时，它支持命名锁，因此任务可以等待给定名称的锁，而不是使用单个全局锁：

import logging
import threading
import time
from functools import wraps
import sys

logger = logging.getLogger(__name__)
# use a global condition for safe manipulating of the LOCKS and
# LOCK_CONDITIONS dictionary in non-atomic operations
GLOBAL_COND = threading.Condition(threading.Lock())
LOCKS = {}
LOCK_CONDITIONS = {}

class ContextDecorator(object):
    def __enter__(self):
        return self

    def __exit__(self, typ, val, traceback):
        pass

    def __call__(self, f):
        @wraps(f)
        def wrapper(*args, **kw):
            with self as acquired:
                if acquired:
                    return f(*args, **kw)
        return wrapper

class TimeoutLock(ContextDecorator):
    def __init__(self, timeout, name=None):
        self.name = name
        self.timeout = timeout

    def __enter__(self):
        with GLOBAL_COND:
            self.cond = LOCK_CONDITIONS.get(self.name, None)
            if self.cond is None:
                self.cond = threading.Condition(threading.Lock())
                LOCK_CONDITIONS[self.name] = self.cond
                LOCKS[self.name] = threading.Lock()
            self.lock = LOCKS[self.name]

        self.cond.acquire()
        current_time = start_time = time.time()
        while current_time < start_time + self.timeout:
            if self.lock.acquire(False):
                self.cond.release()
                return True
            else:
                logger.debug('Waiting')
                self.cond.wait(
                    self.timeout - current_time + start_time)
                logger.debug('Woke up')
                current_time = time.time()
        logger.info('Timed out')
        self.cond.release()
        return False

    def __exit__(self, typ, val, traceback):
        if self.lock.locked():
            self.lock.release()
            with self.cond:
                self.cond.notify_all()


############################# DEMO ###############################
timeout = 4
sleep_interval = 1

handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter(
    fmt=('[%(asctime)s] %(name)s '
         '(%(threadName)s): %(message)s'),
    datefmt='%d/%b/%Y %H:%M:%S'))
logger.addHandler(handler)
logger.setLevel(logging.INFO)

def ascontext(i, name):
    with TimeoutLock(timeout, name=name) as acquired:
        if acquired:
            task()

# this will use a single lock, None
@TimeoutLock(timeout)
def asdecorator(i, name):
    task()

def task():
    logger.info('Acquired')
    time.sleep(sleep_interval)
    logger.info('Released')

def run(target):
    threads = []
    for i, name in enumerate(
            ['foo', 'bar', 'foo', 'baz', 'bar', 'foo']):
        thread = threading.Thread(
            target=target,
            name='{}.{}'.format(name, i),
            args=(i, name))
        threads.append(thread)
        thread.start()
    for i, t in enumerate(threads):
        t.join()


print('---- As context manager ----')
# foo, bar and baz can run concurrently
run(ascontext)
print('---- As decorator ----')
run(asdecorator)

如何在Python 2.7中实现具有超时的锁定

7 个答案: