制片人/消费者多线程

时间:2012-09-23 17:10:54

标签: java multithreading

背景

缺少上学的钱,我正在夜班工作收费站,并利用互联网教自己一些编码技巧,希望明天能有更好的工作或者我在网上销售一些应用程序。漫长的夜晚,很少有客户。

我正在处理多线程作为一个主题,因为我在使用它的文献中遇到了很多代码(例如Android SDK),但我仍然觉得它很模糊。

精神

我现在的方法是:尝试编写我能想到的最基本的多线程示例代码,将我的头撞到墙上一点,然后看看我是否可以拉伸我的大脑来容纳一些新颖的思维方式。我正在暴露自己的极限,希望超越它们。随意批评,挑剔,并指出做我想做的事情的更好方法。

目的

  • Get some advice on how to do the above, based on my efforts so far (code provided)

练习

这是我定义的范围:

定义

创建两个类,这些类在数据对象的生成和消耗方面协同工作。一个 Thread 会创建对象并将其传递到共享空间,以供另一个人接收和使用。让我们调用生成线程Producer,消费线程Consumer和共享空间SharedSpace。产生对象以供另一方消费的行为可以通过类比这种情况来同化:

`Producer`    (a busy mum making chocolate-covered cakes for his child, up to a limit)
`Consumer`    (a hungry child waiting to eat all cakes the mum makes, until told to stop)
`SharedSpace` (a kitchen table on which the cakes are put as soon as they become ready)
`dataValue`   (a chocolate-dripping cake which MUST be eaten immediately or else...)

为了简化练习,我决定允许妈妈在孩子吃蛋糕时做饭。她将等待孩子完成他的蛋糕,然后立即制作另一个,达到一定的限度,以获得良好的养育。这个练习的本质是练习Thread信令,而不是实现任何并发。相反,我专注于完美的序列化,没有投票或者我可以去吗?"检查。我想我必须编写后续练习,其中母亲和孩子一起工作"然后并行。

方法

  • 让我的课程实施 Runnable 界面,以便他们拥有自己的代码入口点

  • 使用我的类作为 Thread 对象的构造函数参数,这些对象是从程序的main入口点

    <实例化并启动的/ LI>
  • 确保main计划在Thread之前未通过 Thread.join()

  • 终止
  • 设置ProducerConsumer

  • 创建数据的次数限制
  • 同意<{>>哨兵值,Produce将用于表示数据生成结束

  • 记录共享资源和数据生成/消费事件锁的获取,包括最终签署工作线程

  • 从程序SharedSpace创建一个main对象,并在开始之前将其传递给每个工作人员

  • 在每个工作人员内部privateSharedSpace对象的引用

  • 提供警示和消息,以描述在生成任何数据之前Consumer准备好消费的情况

  • 在给定次数的迭代后停止Producer

  • 在读取标记值

  • 后停止Consumer

代码


import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
class Consumer extends Threaded {
  public Consumer(SharedSpace sharedSpace) {
    super(sharedSpace);
  }
  @Override
  public void run() {
    super.run();
    int consumedData = 0;
    while (consumedData != -1) {
      synchronized (sharedSpace) {
        logger.info("Acquired lock on sharedSpace.");
        consumedData = sharedSpace.dataValue;
        if (consumedData == 0) {
          try {
            logger.info("Data production has not started yet. "
                + "Releasing lock on sharedSpace, "
                + "until notification that it has begun.");
            sharedSpace.wait();
          } catch (InterruptedException interruptedException) {
            logger.error(interruptedException.getStackTrace().toString());
          }
        } else if (consumedData == -1) {
          logger.info("Consumed: END (end of data production token).");
        } else {
          logger.info("Consumed: {}.", consumedData);
          logger.info("Waking up producer to continue data production.");
          sharedSpace.notify();
          try {
            logger.info("Releasing lock on sharedSpace "
                + "until notified of new data availability.");
            sharedSpace.wait();
          } catch (InterruptedException interruptedException) {
            logger.error(interruptedException.getStackTrace().toString());
          }
        }
      }
    }
    logger.info("Signing off.");
  }
}
class Producer extends Threaded {
  private static final int N_ITERATIONS = 10;
  public Producer(SharedSpace sharedSpace) {
    super(sharedSpace);
  }
  @Override
  public void run() {
    super.run();
    int nIterations = 0;
    while (nIterations <= N_ITERATIONS) {
      synchronized (sharedSpace) {
        logger.info("Acquired lock on sharedSpace.");
        nIterations++;
        if (nIterations <= N_ITERATIONS) {
          sharedSpace.dataValue = nIterations;
          logger.info("Produced: {}", nIterations);
        } else {
          sharedSpace.dataValue = -1;
          logger.info("Produced: END (end of data production token).");
        }
        logger.info("Waking up consumer for data consumption.");
        sharedSpace.notify();
        if (nIterations <= N_ITERATIONS) {
          try {
            logger.info("Releasing lock on sharedSpace until notified.");
            sharedSpace.wait();
          } catch (InterruptedException interruptedException) {
            logger.error(interruptedException.getStackTrace().toString());
          }
        }
      }
    }
    logger.info("Signing off.");
  }
}
class SharedSpace {
  volatile int dataValue = 0;
}
abstract class Threaded implements Runnable {
  protected Logger logger;
  protected SharedSpace sharedSpace;
  public Threaded(SharedSpace sharedSpace) {
    this.sharedSpace = sharedSpace;
    logger = LoggerFactory.getLogger(this.getClass());
  }
  @Override
  public void run() {
    logger.info("Started.");
    String workerName = getClass().getName();
    Thread.currentThread().setName(workerName);
  }
}
public class ProducerConsumer {
  public static void main(String[] args) {
    SharedSpace sharedSpace = new SharedSpace();
    Thread producer = new Thread(new Producer(sharedSpace), "Producer");
    Thread consumer = new Thread(new Consumer(sharedSpace), "Consumer");
    producer.start();
    consumer.start();
    try {
      producer.join();
      consumer.join();
    } catch (InterruptedException interruptedException) {
      interruptedException.printStackTrace();
    }
  }
}

执行日志


Consumer - Started.
Consumer - Acquired lock on sharedSpace.
Consumer - Data production has not started yet. Releasing lock on sharedSpace, until notification that it has begun.
Producer - Started.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 1
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 1.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 2
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 2.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 3
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 3.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 4
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 4.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 5
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 5.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 6
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 6.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 7
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 7.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 8
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 8.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 9
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 9.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: 10
Producer - Waking up consumer for data consumption.
Producer - Releasing lock on sharedSpace until notified.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: 10.
Consumer - Waking up producer to continue data production.
Consumer - Releasing lock on sharedSpace until notified of new data availability.
Producer - Acquired lock on sharedSpace.
Producer - Produced: END (end of data production token).
Producer - Waking up consumer for data consumption.
Producer - Signing off.
Consumer - Acquired lock on sharedSpace.
Consumer - Consumed: END (end of data production token).
Consumer - Signing off.

问题

  • 以上是否正确? (例如,它使用正确的语言工具,正确的方法,它是否包含任何愚蠢的代码,......)

但它看起来是对的&#34;?

即使输出&#34;看起来不错,我也会问正确性。因为你无法想象在我的测试中出现了多少次错误&#34;一次&#34;而不是&#34;另一个&#34; (例如,当消费者首先开始时,制作者在制作哨兵后从未退出时等)。我已经学会了从成功的跑步中获得正确性#34;相反,我对伪并行代码非常怀疑! (根据定义,这个甚至不是平行的!0

扩展答案

一个很好的问题只关注one requested piece of advice(上面的那个),但如果您愿意,请随时提及您对答案中以下其他主题的任何见解:

  • 在下次尝试编码时,如何测试并行代码?

  • 哪些工具可以帮助我进行开发和调试?考虑一下我使用 Eclipse

  • 如果我允许Producer继续生成,那么方法是否会改变,每次生产需要一些可变的时间,而Consumer会消耗任何可用的东西?锁定是否必须移动到其他地方?信号需要从这种等待/通知范式改变吗?

  • 这种做事方法已过时,我宁愿学习别的东西吗?从这个收费站,我不知道在现实世界中发生了什么&#34;

后续步骤

  • 我应该从哪里出发?我已经看到&#34;期货&#34; 在某处提到的概念但我可以使用编号的主题列表按顺序进行处理,通过教学方式进行订购相关的学习资源

Tino Sino

4 个答案:

答案 0 :(得分:6)

  

以上是否正确?

我看到的唯一问题是@Tudor和@Bhaskar提到的问题。无论何时在等待时测试某个条件,必须使用while循环。然而,这更多是关于与多个生产者和消费者的竞争条件。可能会发生虚假的唤醒,但竞争条件更有可能发生。见my page on the subject

是的,您只有1个生产者和1个消费者,但您可能会尝试为多个消费者扩展代码或将代码复制到另一个方案。

  

我已经学会了不要从“成功的运行”中声称正确性。相反,我对伪并行代码非常怀疑!

本能很好。

  

我在下次尝试编码时如何测试并行代码?

这很难。扩展它是一种方式。添加多个生产者和消费者,看看是否有问题。在具有不同数量/类型的处理器的多个体系结构上运行。您最好的防御将是代码正确性。紧密同步,充分利用BlockingQueueExecutorService等课程,让您更贴近/更清洁。

没有简单的答案。测试多线程代码非常困难。

  

哪些工具可以帮助我进行开发和调试?

就一般内容而言,我会研究像Emma这样的覆盖工具,这样您就可以确保您的单元测试覆盖了所有代码。

就多线程代码测试而言,了解如何阅读kill -QUIT线程转储并查看Jconsole内部运行的线程。像YourKit这样的Java分析器也可以提供帮助。

  

如果我允许制作人继续制作,那么这种方法是否会改变,每次制作都会花费不同的时间......

我不这么认为。消费者将永远等待制片人。也许我不理解这个问题?

  

这种做事方法是否过时,我宁愿学习别的东西吗?从这个收费站,我不知道“在Java的现实世界中发生了什么”

下一步了解ExecutorService classes。它们处理大部分new Thread()样式代码 - 特别是当您处理使用线程执行的许多异步任务时。这是一个tutorial

  

我应该从哪里离开?

再次,ExecutorService。我假设您已阅读this starting docs。正如@Bhaskar所说,Java Concurrency in Practice是一本很好的圣经。


以下是有关您的代码的一般性评论:

  • SharedSpaceThreaded类似乎是一种人为的方法。如果你正在玩基类之类的话就好了。但总的来说,我从不使用这样的模式。生产者和消费者通常使用类似LinkedBlockingQueueBlockingQueue,在这种情况下,同步和volatile有效负载将由您负责。此外,我倾向于将共享信息注入到对象构造函数中,而不是从基类中获取它。

  • 通常,如果我使用的是synchronized,则它位于private final字段上。我经常创建一个private final Object lockObject = new Object();用于锁定,除非我已经使用了一个对象。

  • 注意巨大的synchronized块并将日志消息放入synchronized部分。日志通常对文件系统执行synchronized IO,这可能非常昂贵。如果可能的话,你应该有一个小的,非常紧的synchronized块。

  • 您在循环外定义consumedData。我会在赋值时定义它,然后使用break从循环中保释,如果它是== -1。确保尽可能限制局部变量范围。

  • 您的日志消息将主导您的代码性能。这意味着当您删除它们时,您的代码将以不同方式执行完全。当您使用它调试问题时,这非常非常重要。当您迁移到具有不同CPU /核心的不同架构时,性能(很可能)也会发生变化。

  • 您可能知道这一点,但是当您致电sharedSpace.notify();时,这只意味着另一个线程会被通知,如果当前位于sharedSpace.wait();。如果它不是别的,那么它将错过通知。仅供参考。

  • 做一个if (nIterations <= N_ITERATIONS)有点奇怪,然后else下面的3行再做一次。复制notify()会更好地简化分支。

  • 然后在++内有int nIterations = 0;然后while。这是一个for循环的配方:

    for (int nIterations = 0; nIterations <= N_ITERATIONS; nIterations++) {
    

这是一个更严格的代码版本。这只是我如何写它的一个例子。同样,除了缺失的while之外,您的版本似乎没有任何问题。

public class Consumer implements Runnable {
    private final BlockingQueue<Integer> queue;
    public Consumer(BlockingQueue<Integer> queue) {
       this.queue = queue;
    }
    @Override
    public void run() {
       while (true) {
          int consumedData = queue.take();
          if (consumedData ==  Producer.FINAL_VALUE) {
              logger.info("Consumed: END (end of data production token).");
              break;
          }
          logger.info("Consumed: {}.", consumedData);
       }
       logger.info("Signing off.");
    }
}

public class Producer implements Runnable {
    public static final int FINAL_VALUE = -1;
    private final BlockingQueue<Integer> queue;
    public Producer(BlockingQueue<Integer> queue) {
       this.queue = queue;
    }
    @Override
    public void run() {
       for (int nIterations = 0; nIterations <= N_ITERATIONS; nIterations++) {
          logger.info("Produced: {}", nIterations);
          queue.put(nIterations);
       }
       queue.put(FINAL_VALUE);
       logger.info("Produced: END (end of data production token).");
       logger.info("Signing off.");
    }
}

public class ProducerConsumer {
    public static void main(String[] args) {
       // you can add an int argument to the LinkedBlockingQueue constructor
       // to only allow a certain number of items in the queue at one time
       BlockingQueue<Integer> queue = new LinkedBlockingQueue<Integer>();
       Thread producer = new Thread(new Producer(queue), "Producer");
       Thread consumer = new Thread(new Consumer(queue), "Consumer");
       // start and join go here
    }
}

答案 1 :(得分:4)

你似乎在这里做得很好。实际上并没有太多的挑剔。有人认为我想推荐你应该避免在缓冲对象本身上进行同步。在这种情况下它没关系,但假设您切换到数据结构缓冲区,取决于它可能在内部同步的类(例如Vector,虽然它现在已经过时),因此从外部获取锁定可能会弄乱它了。

编辑:Bhaskar提到了使用while来回复wait的问题。这是因为可能发生的臭名昭着的虚假唤醒,迫使线程过早地从wait出来,所以你需要确保它重新进入。

接下来你要做的是实现有限缓冲区生产者消费者:拥有一些共享数据结构,例如:链表并设置最大尺寸(例如10个项目)。然后让生产者继续生产,只有当队列中有10个项目时才暂停它。只要缓冲区为空,消费者就会被暂停。

您可以采取的后续步骤是学习如何自动化您手动实施的流程。看一下提供阻塞行为缓冲区的BlockingQueue(即如果缓冲区为空,则消费者将自动阻止,如果填充程序已满,则生产者将阻止。)

此外,根据具体情况,执行者(查看ExecutorService)可能是一个有价值的替代品,因为它们封装了一个任务队列和一个或多个工作者(消费者),因此您只需要生产者。

答案 2 :(得分:0)

ProducerConsumer可以是实现Runnable的简单类(无extends Threaded)这样它们就不那么脆弱了。客户端可以创建Thread个自己并附加实例,因此不需要类层次结构的开销。

wait()之前的情况应该是while()而不是if

编辑:来自JCIP第301页:

void stateDependentMethod() throws InterruptedException {
      // condition predicate must be guarded by lock
      synchronized(lock) {
          while (!conditionPredicate())
            lock.wait();
          // object is now in desired state
       }
  }

你已经建立了静止停止的条件。通常情况下,生产者和消费者应该更加灵活 - 他们应该能够响应外部信号停止。

首先,为了实现外部停止信号,你有一个标志:

class Producer implements Runnable { 
     private volatile boolean stopRequested ;

     public void run() {
        while(true){
           if(stopRequested )
                // get out of the loop
         }
     }

     public void stop(){
        stopRequested  = true;
        // arrange to  interrupt the Producer thread here.
     }
 }

当您尝试实施上述内容时,您可能会发现还会出现其他并发症 - 例如 - 您的制作人首先发布然后wait(),但这可能会导致问题。

如果您有兴趣进一步阅读,我建议您阅读本书 - Java Concurrency In Practice。这将有很多建议,而不是我在这里添加的。

答案 3 :(得分:0)

雄心勃勃!您在8年前问过这个问题。我希望您的努力为您提供了(并继续为您提供)您想要的教育。

强烈建议不要使用wait()notify()join()这几天来在Java中实现多线程。当您尝试以较低的级别控制并发时,实在太容易了(事实上Java设计人员承认Thread的许多方法和语义实际上是设计错误,但是他们不得不将它们留在出于向后兼容的目的-许多人不再使用新的“虚拟线程”(Project Loom)-但这是一个不同的主题。

当今,手动启动和控制线程的首选方法是通过ExecutorService.submit(Callable<V>),返回一个Future<V>。然后,您可以通过调用Future<V>.get(),返回由可调用对象返回的类型V的值来等待线程退出(并获取返回值)(如果ExecutionException引发了一个未捕获的异常。

以下类是如何实现此类示例的示例。这将通过一个有限的阻塞队列将任意数量的生产者连接到任意数量的消费者。 (来自线程的返回值被忽略,因此调用Callable,返回ExecutorService.submit(Runnable)而不是Future<?>。)

ExecutorService.submit(Callable<V>)

用法如下:

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.Callable;
import java.util.concurrent.CancellationException;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

public abstract class ProducerConsumer<E> {

    private final BlockingQueue<Optional<E>> queue;

    public ProducerConsumer(
            int numProducerThreads, int numConsumerThreads, int queueCapacity) {
        if (numProducerThreads < 1 || numConsumerThreads < 1 || queueCapacity < 1) {
            throw new IllegalArgumentException();
        }
        queue = new ArrayBlockingQueue<Optional<E>>(queueCapacity);
        final ExecutorService executor = 
                Executors.newFixedThreadPool(numProducerThreads + numConsumerThreads);
        try {
            // Start producer threads
            final List<Future<?>> producerFutures = new ArrayList<>();
            final AtomicInteger numLiveProducers = new AtomicInteger();
            for (int i = 0; i < numProducerThreads; i++) {
                producerFutures.add(executor.submit(() -> {
                    numLiveProducers.incrementAndGet();
                    // Run producer
                    producer();
                    // When last producer finishes, deliver poison pills to consumers
                    if (numLiveProducers.decrementAndGet() == 0) {
                        for (int j = 0; j < numConsumerThreads; j++) {
                            queue.put(Optional.empty());
                        }
                    }
                    return null;
                }));
            }
            // Start consumer threads
            final List<Future<?>> consumerFutures = new ArrayList<>();
            for (int i = 0; i < numConsumerThreads; i++) {
                consumerFutures.add(executor.submit(() -> {
                    // Run Consumer
                    consumer();
                    return null;
                }));
            }
            // Wait for all producers to complete
            completionBarrier(producerFutures, false);
            // Shut down any consumers that are still running after producers complete
            completionBarrier(consumerFutures, false);
        } finally {
            executor.shutdownNow();
        }
    }

    private static void completionBarrier(List<Future<?>> futures, boolean cancel) {
        for (Future<?> future : futures) {
            try {
                if (cancel) {
                    future.cancel(true);
                }
                future.get();
            } catch (CancellationException | InterruptedException e) {
                // Ignore
            } catch (ExecutionException e) {
                throw new RuntimeException(e);
            }
        }
    }

    protected void produce(E val) {
        try {
            queue.put(Optional.of(val));
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    protected Optional<E> consume() {
        try {
            return queue.take();
        } catch (InterruptedException e) {
            throw new RuntimeException(e);
        }
    }

    /** Producer loop. Call {@link #produce(E)} for each element. */
    public abstract void producer();

    /**
     * Consumer thread. Call {@link #consume()} to get each successive element,
     * until an empty {@link Optional} is returned.
     */
    public abstract void consumer();
}