Question

我通过asyncio.subprocess运行外部下载程序脚本，每当我尝试下载大数据时asyncio都会出现以下错误：

asyncio.streams.LimitOverrunError：找不到分隔符和块超过限额

是什么导致这种情况，我该如何解决？

import asyncio, subprocess, websockets, json
from os.path import expanduser, sep

async def handler(websocket, path):
    print("New client connected.")
    await websocket.send('CONNECTED')
    path = expanduser("~") + sep
    try:
        while True:
            inbound = await websocket.recv()
            if inbound is None:
                break
            while inbound != None:
                cmd = ('downloader_script', '-v', '-p', '-o', '/home/blah/blah', inbound)
                process = await asyncio.create_subprocess_exec(*cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
                async for output in process.stdout:
                    for line in output.decode().split('\r'):
                        line = line.strip()
                        if line == '':
                            continue
                        data = {}
                        await asyncio.sleep(1)
                        if line.startswith('INFO:'):
                            data['INFO'] = line.split('INFO: ')[1]
                        elif line.startswith('['):
                            data['progress'] = line.split(']')[0][1:]
                        elif line.startswith('ERROR:'):
                            data['ERROR'] = line.split('ERROR: ')[1]
                        else:
                                data['message'] = line
                        print (data)
                        await websocket.send(json.dumps(data))
                await websocket.send(json.dumps({'progress': 'DONE'}))
                await websocket.send('bye!')
                break
    except websockets.exceptions.ConnectionClosed:
        print("Client disconnected.")

if __name__ == "__main__":
    server = websockets.serve(handler, '0.0.0.0', 8080)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(server)
    loop.run_forever()

Answer 1

由于asyncio.subprocess目前最适合使用换行符，因此可以尝试使用\r代替\n来运行脚本。

这是一个可以提供帮助的小型bash脚本（cr2lf.sh）：

#!/bin/bash
$@ | tr '\r' '\n'

示例：

# slow_ugly_writer.py
import sys
import time

n = int(sys.argv[1])

for i in range(1, n + 1):
    time.sleep(0.5)
    print("*" * i, end="\r")

用法：

./cr2lf.sh python -u slow_ugly_writer.sh 10

从你的程序中：

cmd = ('crlf.sh', 'downloader_script', '-v', '-p', '-o', '/home/blah/blah', inbound)

ValueError：找不到分隔符，并且块超出限制

1 个答案: