Question

我正在尝试使用多个或并行的消费者来实现生产者/消费者模式。

我做了一个实现，但我想知道它有多好。有人可以做得更好吗？你们有没有发现任何错误？

不幸的是我不能使用TPL数据流，因为我们在项目的最后并且在我们的包中加入一个额外的库会花费大量的文书工作而我们没有那么多时间。

我想做的是加快以下部分：

anIntermediaryList = StepOne(anInputList); // I will put StepOne as Producer :-) Step one is remote call.

aResultList = StepTwo(anIntermediaryList); // I will put StepTwo as Consumer, however he also produces result. Step two is also a remote call.
// StepOne is way faster than StepTwo.

为此我想出了我将输入列表（anInputList）

的块

StepOne将位于Producer内部，并将中间块放入队列中。将有多个生产者，他们将采取中间结果并使用StepTwo处理它。

以下是该实施的简化版本：

Task.Run(() => {
     aChunkinputList = Split(anInputList)
     foreach(aChunk in aChunkinputList)
     {
          anIntermediaryResult = StepOne(aChunk)
          intermediaryQueue.Add(anIntermediaryResult)
     }
})

while(intermediaryQueue.HasItems)
{
     anItermediaryResult = intermediaryQueue.Dequeue()
     Task.Run(() => {
         aResultList = StepTwo(anItermediaryResult);
         resultQueue.Add(aResultList)
     }
}

我还认为并行运行的消费者的最佳数字是：＆＃34; Environment.ProcessorCount / 2＆＃34;。我想知道这是否也是一个好主意。

现在这是我的模拟实现，问题是有人可以做得更好或发现任何错误吗？

class Example
{
    protected static readonly int ParameterCount_ = 1000;
    protected static readonly int ChunkSize_ = 100;
    // This might be a good number for the parallel consumers.
    protected static readonly int ConsumerCount_ = Environment.ProcessorCount / 2;
    protected Semaphore mySemaphore_ = new Semaphore(Example.ConsumerCount_, Example.ConsumerCount_);

    protected ConcurrentQueue<List<int>> myIntermediaryQueue_ = new ConcurrentQueue<List<int>>();
    protected ConcurrentQueue<List<int>> myResultQueue_ = new ConcurrentQueue<List<int>>();

    public void Main()
    {
        List<int> aListToProcess = new List<int>(Example.ParameterCount_ + 1);
        aListToProcess.AddRange(Enumerable.Range(0, Example.ParameterCount_));

        Task aProducerTask = Task.Run(() => Producer(aListToProcess));

        List<Task> aTaskList = new List<Task>();            
        while(!aProducerTask.IsCompleted || myIntermediaryQueue_.Count > 0)
        {
            List<int> aChunkToProcess;
            if (myIntermediaryQueue_.TryDequeue(out aChunkToProcess))
            {
                mySemaphore_.WaitOne();
                aTaskList.Add(Task.Run(() => Consumer(aChunkToProcess)));
            }
        }

        Task.WaitAll(aTaskList.ToArray());

        List<int> aResultList = new List<int>();
        foreach(List<int> aChunk in myResultQueue_)
        {
            aResultList.AddRange(aChunk);
        }
        aResultList.Sort();
        if (aListToProcess.SequenceEqual(aResultList))
        {
            Console.WriteLine("All good!");
        }
        else
        {
            Console.WriteLine("Bad, very bad!");
        }
    }

    protected void Producer(List<int> elements_in)
    {
        List<List<int>> aChunkList = Example.SplitList(elements_in, Example.ChunkSize_);

        foreach(List<int> aChunk in aChunkList)
        {
            Console.WriteLine("Thread Id: {0} Producing from: ({1}-{2})", 
                Thread.CurrentThread.ManagedThreadId,
                aChunk.First(),
                aChunk.Last());

            myIntermediaryQueue_.Enqueue(ProduceItemsRemoteCall(aChunk));
        }
    }

    protected void Consumer(List<int> elements_in)
    {
        Console.WriteLine("Thread Id: {0} Consuming from: ({1}-{2})",
                Thread.CurrentThread.ManagedThreadId,
                Convert.ToInt32(Math.Sqrt(elements_in.First())),
                Convert.ToInt32(Math.Sqrt(elements_in.Last())));

        myResultQueue_.Enqueue(ConsumeItemsRemoteCall(elements_in));
        mySemaphore_.Release();
    }

    // Dummy Remote Call
    protected List<int> ProduceItemsRemoteCall(List<int> elements_in)
    {
        return elements_in.Select(x => x * x).ToList();
    }

    // Dummy Remote Call
    protected List<int> ConsumeItemsRemoteCall(List<int> elements_in)
    {
        return elements_in.Select(x => Convert.ToInt32(Math.Sqrt(x))).ToList();
    }

    public static List<List<int>> SplitList(List<int> masterList_in, int chunkSize_in)
    {
        List<List<int>> aReturnList = new List<List<int>>();
        for (int i = 0; i < masterList_in.Count; i += chunkSize_in)
        {
            aReturnList.Add(masterList_in.GetRange(i, Math.Min(chunkSize_in, masterList_in.Count - i)));
        }
        return aReturnList;
    }
}

主要功能：

class Program
{
    static void Main(string[] args)
    {
        Example anExample = new Example();
        anExample.Main();
    }
}

再见的Laszlo

Answer 1

根据评论，我发布了第二个和第三个版本： https://codereview.stackexchange.com/questions/71182/producer-consumer-in-c-with-multiple-parallel-consumers-and-no-tpl-dataflow/71233#71233

具有多个（并行）使用者且没有TPL数据流的C＃中的生产者消费者

1 个答案: