Question

最初我有一个小脚本使用本机Python线程池执行一些IO绑定作业，例如

from multiprocessing.dummy import Pool as ThreadPool

def f():
    # some IO bound action here.
    pass


def pooled_f():
    pool = ThreadPool(4)
    args = [1, 2, 3, 4]
    results = []
    for i in pool.imap_unordered(f, args):
        results.append(i)
    pool.close()
    pool.join()
    return results

现在我正在使用Flask + Celery + Redis来允许用户向工作人员提交不同的工作并有效地执行：用户提交包含一些数据的文件，芹菜会做一些工作，并向用户展示。也许这对于这么小的应用来说太过分了，但它无论如何都应该有用。

@celery.task(bind=True)
def sillywork(self, filename):
    filesize = os.path.getsize(filename)
    with open(filename, 'rb') as f:
        chunked = 0
        for i, line in enumerate(f):
            message = line.strip()
            chunked += len(message)
            # print line
            self.update_state(
                state='PROGRESS',
                meta={
                    'current': chunked,
                    'total': filesize,
                    'status': message
                })

            time.sleep(0.01)

    return {
        'current': filesize,
        'total': filesize,
        'status': 'OK',
        'result': 1
    }




@app.route('/jobs', methods=['POST', 'GET'])
def jobs():
    form = Form()
    if request.method == 'POST':
        filename = form.data.filename
        # file contains the data that user submitted
        file = request.files['file']
        file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
        # How should I add the celery job here?
        sillywork.apply_async(args=[filename])
        return redirect(url_for('index', task_id=task.id))

我按照https://blog.miguelgrinberg.com/post/using-celery-with-flask示例，不知怎的，我找不到让芹菜完成我需要的工作的方法。

如何组合使用Python线程池和芹菜？

0 个答案: