Question

使用python watchdog文件系统事件观察库我注意到在Windows Server 2003下使用时，它进入了＆＃34;轮询模式＆＃34;因此，停止使用异步操作系统通知，因此，在大量文件更改时会严重降低系统性能。

我将问题追溯到使用watchdog/observers/winapi.py系统调用的CancelIoEx文件，以便在用户想要停止监视被监视的目录或文件时停止ReadDirectoryChangesW调用锁定：

（winapi.py）

CancelIoEx = ctypes.windll.kernel32.CancelIoEx
CancelIoEx.restype = ctypes.wintypes.BOOL
CancelIoEx.errcheck = _errcheck_bool
CancelIoEx.argtypes = (
    ctypes.wintypes.HANDLE,  # hObject
    ctypes.POINTER(OVERLAPPED)  # lpOverlapped
)

...
...
...

def close_directory_handle(handle):
    try:
        CancelIoEx(handle, None)  # force ReadDirectoryChangesW to return
    except WindowsError:
        return

CancelIoEx调用的问题是它在Windows Server 2008之前不可用： http://msdn.microsoft.com/en-us/library/windows/desktop/aa363792(v=vs.85).aspx

一种可能的替代方法是更改close_directory_handle以使其在受监控目录中创建模拟文件，从而解锁等待ReadDirectoryChangesW返回的线程。

但是，我注意到Windows Server 2003中的CancelIo系统调用是in fact available：

取消发出的所有待处理输入和输出（I / O）操作通过指定文件的调用线程。该功能没有取消其他线程为文件句柄发出的I / O操作。至从另一个线程取消I / O操作，使用CancelIoEx 功能

但是调用CancelIo不会影响等待的线程。

您对如何解决此问题有任何想法吗？可以使用threading.enumerate()发出一个信号，由从这些处理程序调用的每个线程CancelIo处理？

Answer 1

自然的方法是实现一个完成例程，并使用其重叠模式调用ReadDirectoryChangesW。以下示例显示了执行此操作的方法：

RDCW_CALLBACK_F = ctypes.WINFUNCTYPE(None, ctypes.wintypes.DWORD, ctypes.wintypes.DWORD, ctypes.POINTER(OVERLAPPED))

首先，创建一个WINFUNCTYPE工厂，用于生成（可从Windows API调用）C类似python方法的函数。在这种情况下，没有返回值和3个参数对应

VOID CALLBACK FileIOCompletionRoutine(
  _In_     DWORD dwErrorCode,
  _In_     DWORD dwNumberOfBytesTransfered,
  _Inout_  LPOVERLAPPED lpOverlapped
);

FileIOCompletionRoutine标题。

需要将回调引用以及重叠结构添加到ReadDirectoryChangesW参数列表中：

ReadDirectoryChangesW = ctypes.windll.kernel32.ReadDirectoryChangesW

ReadDirectoryChangesW.restype = ctypes.wintypes.BOOL
ReadDirectoryChangesW.errcheck = _errcheck_bool
ReadDirectoryChangesW.argtypes = (
    ctypes.wintypes.HANDLE,  # hDirectory
    LPVOID,  # lpBuffer
    ctypes.wintypes.DWORD,  # nBufferLength
    ctypes.wintypes.BOOL,  # bWatchSubtree
    ctypes.wintypes.DWORD,  # dwNotifyFilter
    ctypes.POINTER(ctypes.wintypes.DWORD),  # lpBytesReturned
    ctypes.POINTER(OVERLAPPED),  # lpOverlapped
    RDCW_CALLBACK_F  # FileIOCompletionRoutine # lpCompletionRoutine
)

从这里开始，我们准备执行重叠的系统调用。这是一个简单的调用bacl，只是用来测试一切正常：

def dir_change_callback(dwErrorCode,dwNumberOfBytesTransfered,p):
     print("dir_change_callback! PID:" + str(os.getpid()))
     print("CALLBACK THREAD: " + str(threading.currentThread()))

准备并执行电话：

event_buffer = ctypes.create_string_buffer(BUFFER_SIZE)
nbytes = ctypes.wintypes.DWORD()
overlapped_read_dir = OVERLAPPED()
call2pass = RDCW_CALLBACK_F(dir_change_callback)

hand = get_directory_handle(os.path.abspath("/test/"))

def docall():
    ReadDirectoryChangesW(hand, ctypes.byref(event_buffer),
                          len(event_buffer), False,
                          WATCHDOG_FILE_NOTIFY_FLAGS,
                          ctypes.byref(nbytes), 
                          ctypes.byref(overlapped_read_dir), call2pass)

print("Waiting!")
docall()

如果您将所有这些代码加载并执行到DreamPie交互式shell中，您可以检查系统调用是否完成以及回调是否执行，从而在{{1}下完成第一次更改后打印线程和pid编号} 目录。此外，您会注意到它们与主线程和进程相同：尽管事件是由一个独立的线程引发的，但回调在与我们的主程序相同的进程和线程中运行，因此提供了一个不希望的行为：

c:\test

该程序将锁定主线程，并且回调将永远不会执行。我尝试了很多同步工具，甚至Windows API信号量总是得到相同的行为，所以最后，我决定使用lck = threading.Lock() def dir_change_callback(dwErrorCode,dwNumberOfBytesTransfered,p): print("dir_change_callback! PID:" + str(os.getpid())) print("CALLBACK THREAD: " + str(threading.currentThread())) ... ... ... lck.acquire() print("Waiting!") docall() lck.acquire()的同步配置在使用ReadDirectoryChangesW python管理和同步的单独进程中实现异步调用库：

对multiprocessing的调用不会返回由Windows API提供的句柄号，而是由get_directory_handle库管理的句号，因为我实现了句柄生成器：

winapi

每个生成的句柄必须与文件系统路径全局关联：

class FakeHandleFactory():
    _hl = threading.Lock()
    _next = 0
    @staticmethod
    def next():
        FakeHandleFactory._hl.acquire()
        ret = FakeHandleFactory._next
        FakeHandleFactory._next += 1
        FakeHandleFactory._hl.release()
        return ret

每次调用handle2file = {}现在都会生成read_directory_changes（来自ReadDirectoryRequest）对象：

multiprocessing.Process

此类指定class ReadDirectoryRequest(multiprocessing.Process): def _perform_and_wait4request(self, path, recursive, event_buffer, nbytes): hdl = CreateFileW(path, FILE_LIST_DIRECTORY, WATCHDOG_FILE_SHARE_FLAGS, None, OPEN_EXISTING, WATCHDOG_FILE_FLAGS, None) #print("path: " + path) aux_buffer = ctypes.create_string_buffer(BUFFER_SIZE) aux_n = ctypes.wintypes.DWORD() #print("_perform_and_wait4request! PID:" + str(os.getpid())) #print("CALLBACK THREAD: " + str(threading.currentThread()) + "\n----------") try: ReadDirectoryChangesW(hdl, ctypes.byref(aux_buffer), len(event_buffer), recursive, WATCHDOG_FILE_NOTIFY_FLAGS, ctypes.byref(aux_n), None, None) except WindowsError as e: print("!" + str(e)) if e.winerror == ERROR_OPERATION_ABORTED: nbytes = 0 event_buffer = [] else: nbytes = 0 event_buffer = [] # Python 2/3 compat nbytes.value = aux_n.value for i in xrange(self.int_class(aux_n.value)): event_buffer[i] = aux_buffer[i] CloseHandle(hdl) try: self.lck.release() except: pass def __init__(self, handle, recursive): buffer = ctypes.create_string_buffer(BUFFER_SIZE) self.event_buffer = multiprocessing.Array(ctypes.c_char, buffer) self.nbytes = multiprocessing.Value(ctypes.wintypes.DWORD, 0) targetPath = handle2file.get(handle, None) super(ReadDirectoryRequest, self).__init__(target=self._perform_and_wait4request, args=(targetPath, recursive, self.event_buffer, self.nbytes)) self.daemon = True self.lck = multiprocessing.Lock() self.result = None try: self.int_class = long except NameError: self.int_class = int if targetPath is None: self.result = ([], -1) def CancelIo(self): try: self.result = ([], 0) self.lck.release() except: pass def read_changes(self): #print("read_changes! PID:" + str(os.getpid())) #print("CALLBACK THREAD: " + str(threading.currentThread()) + "\n----------") if self.result is not None: raise Exception("ReadDirectoryRequest object can be used only once!") self.lck.acquire() self.start() self.lck.acquire() self.result = (self.event_buffer, self.int_class(self.nbytes.value)) return self.result提供执行系统调用并等待（或）的进程：

已提出更改事件。
主线程通过调用Process对象ReadDirectoryRequest方法取消请求。

请注意：

get_directory_handle
close_directory_handle
read_directory_changes

角色现在要管理请求。为此，需要线程锁和辅助数据结构：

CancelIo

<强> get_directory_handle

rqIndexLck = threading.Lock() # Protects the access to `rqIndex`
rqIndex = {} # Maps handles to request objects sets.

<强> close_directory_handle

def get_directory_handle(path):
    rqIndexLck.acquire()
    ret = FakeHandleFactory.next()
    handle2file[ret] = path
    rqIndexLck.release()
    return ret

最后但并非最不重要： read_directory_changes

def close_directory_handle(handle):
    rqIndexLck.acquire()
    rqset4handle = rqIndex.get(handle, None)
    if rqset4handle is not None:
        for rq in rqset4handle:
            rq.CancelIo()
        del rqIndex[handle]
    if handle in handle2file:
        del handle2file[handle]
    rqIndexLck.release()

看门狗兼容性：“CancelIoEx”的解决方法

1 个答案: