Question

我正在尝试在python3.6中异步执行许多阻止任务。所有阻止任务都将数据存储在SQLITE3（peewee orm）中。尽管执行这些任务有时会给我sqlite3的DB Block错误。我正在使用sanic，Peewee和python3.6。

任何可解决此DBBlock错误的工作或改进代码。

#sqlite3_store_func is a blocking sqlite3 store function, which insert #data into the database

async def function_a(config, loop, executor):

    ins = SomesyncClass(path, config)

    ##this gives a list of data
    _, purchases = await ins.parse()

    #Blocking functions should not be called directly. 
    #For example, if a function blocks for 1 second, other tasks are
    # delayed by 1 second which can have an important impact on
    # reactivity.
    # An executor can be used to run a task in a different thread or 
    #even in a different process, to not block the thread of the 
    #event loop. See the AbstractEventLoop.run_in_executor() method. 
    await asyncio.wait(
            fs=[loop.run_in_executor(executor,  
                functools.partial(sqlite3_store_func, **purchase)) for purchase in purchases],
            return_when=asyncio.ALL_COMPLETED)
    return


async def parse(config, loop, executor):

    await asyncio.gather(*[
                function_a(config, loop, executor), 
                function_b(config, loop, executor),
                ])

    logger.info('Periodic task has finished execution')

@SANIC_BLUEPRINT.post('parse')
async def parse(request):
    """
    To get all the assets created by the requester
    """
    request.app.config.VALIDATE_FIELDS(["path"], request.json)

    #loop = asyncio.get_event_loop()
    executor = concurrent.futures.ThreadPoolExecutor(max_workers=5)

    ##task which is non blocking
    request.app.add_task(parse_takeout(request.app.config, request.app.loop, executor))
    return response.json({"error": True, "success": False})

Answer 1

几件事要注意。

Sqlite嵌入到python进程中。它不是通过套接字与之通信的单独服务器，因此已经不像您将能够利用事件循环来异步查询sqlite db。

这使您可以使用线程池。众所周知，sqlite使用全局写锁定，因此在任何给定时间只能有一个连接写到数据库。这意味着，如果您正在使用线程池，则要么需要在写操作周围放置一个互斥体，要么对所有写操作使用专用线程，要么从容地处理获取锁的失败。

在异步python中运行sqlite插入操作

1 个答案: