Question

import csv
from itertools import groupby
import requests
from gevent.threadpool import ThreadPoolExecutor

def postRequest():
    postapi = requests.post(ENDPOINT_URL, json=groups, headers=headers)


pool = ThreadPoolExecutor(max_workers=20)

with open("car.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [dict(d) for d in r]
    groups = {}

    for k, g in groupby(data, lambda r: (r[ 'bmw' ])):
        items = [ ]
        for i in g:
           #data process

        pool.submit(postRequest)

pool.shutdown(wait=True)

我的代码运行正常，但是问题是某些数据将被复制。我认为发生重复的线程会在同一时间发布相同的请求，因为它是并发方法。我可以知道如何避免这种情况吗？

已更新

def postRequest(message):
    postapi = requests.post(ENDPOINT_URL, json=groups, headers=headers)
    return message


pool = ThreadPoolExecutor(max_workers=20)

with open("car.csv", "r") as csv_ledger:
    r = csv.DictReader(csv_ledger)
    data = [dict(d) for d in r]
    groups = {}

    for k, g in groupby(data, lambda r: (r[ 'bmw' ])):
        items = [ ]
        for i in g:
           #data process

        future = pool.submit(postRequest,("hello"))
        print(future.done())
        sleep(2)
        print(future.result())

pool.shutdown(wait=True)

Answer 1

submit函数采用应传递给该函数的参数。也许尝试让它们通过？

    pool.submit(postRequest, x, y, z)

为什么在使用并发线程时数据重复

1 个答案: