Question

我正在尝试在以下场景中找到使用ThreadPoolExecutor的方法：

我有一个单独的线程在线程池上生成和提交任务
任务提交是同步的，并将阻止，直到ThreadPoolExecutor
在任何给定时间，只能执行固定数量的任务并行执行。同时运行的无限数量的任务可能会导致内存耗尽。
在提交任务之前，生产者线程始终检查自第一个提交的任务以来未超过某个最大构建时间。如果超出，则线程将关闭，但当前在线程池上运行的任何任务在应用程序终止之前都会运行完毕。
当生产者线程终止时，线程池的队列上不应该有任何未启动的任务。

为了提供更多上下文，我目前只提交所有任务，并在最长构建时间到期后取消ExecutorService.submit返回的所有期货。我忽略了所有结果CancellationException，因为它们是预期的。问题是Future.cancel(false)的行为很奇怪且不适合我的用例：

它可以阻止任何未启动的任务运行（好）
它不会中断当前正在运行的任务并让它们运行完成（好）
但是，它会忽略当前正在运行的任务抛出的任何异常，而是抛出CancellationException为Exception.getCause() null的{{1}}。因此，我无法区分在运行之前已取消的任务与在最长构建时间之后继续运行且由于异常而失败的任务！这很不幸，因为在这种情况下我想传播异常并将其报告给一些错误处理机制。

我查看了Java提供的不同阻塞队列，并发现了这一点：https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/SynchronousQueue.html。这看起来似乎很理想，但随后看https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html，它似乎与ThreadPoolExecutor一样，不像我想要的那样：

直接切换。工作队列的一个很好的默认选择是a SynchronousQueue将任务交给线程，否则抱着他们。在这里，如果没有线程，尝试排队任务将失败立即可以运行它，所以一个新的线程将是建造。此策略在处理请求集时避免了锁定可能有内部依赖关系。一般直接切换要求无限制的maximumPoolSizes以避免拒绝新提交的内容任务。这反过来也承认了无限线程增长的可能性当命令继续平均到达的速度超过它们的速度时处理。

理想情况是SynchronousQueue.poll上的消费者（=池）阻塞和SynchronousQueue.put上的生产者（=任务生成者线程）阻塞。

我知道如何在不编写任何复杂的调度逻辑的情况下实现我描述的场景（ThreadPoolExecutor应该包含哪些内容）？

Answer 1

我相信你走在了正确的道路上......你所要做的就是使用SynchronousQueue与RejectedExecutionHandler结合，使用以下constructor ......通过这种方式，您可以定义一个固定的最大大小线程池（限制您的资源使用）并定义一个回退机制来重新安排那些无法处理的任务（因为该池已满）......例如：

public class Experiment {

    public static final long HANDLER_SLEEP_TIME = 4000;
    public static final int MAX_POOL_SIZE = 1;

    public static void main(String[] args) throws InterruptedException {
        SynchronousQueue<Runnable> queue;
        RejectedExecutionHandler handler;
        ThreadPoolExecutor pool;
        Runnable runA, runB;

        queue   = new SynchronousQueue<>();
        handler = new RejectedExecutionHandler() {
            @Override
            public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
                try {
                    System.out.println("Handler invoked! Thread: " + Thread.currentThread().getName());
                    Thread.sleep(HANDLER_SLEEP_TIME); // this let runnableA finish
                    executor.submit(r);    // re schedule

                } catch (InterruptedException ex) {
                    throw new RuntimeException("Handler Exception!", ex);
                }
            }
        };

        pool = new ThreadPoolExecutor(1, MAX_POOL_SIZE, 10, TimeUnit.SECONDS, queue, handler);
        runA = new Runnable() {
            @Override
            public void run() {
                try {
                    Thread.sleep(3000);
                    System.out.println("hello, I'm runnable A");

                } catch (Exception ex) {
                    throw new RuntimeException("RunnableA", ex);
                }
            }
        };
        runB = new Runnable() {
            @Override
            public void run() {
                System.out.println("hello, I'm runnable B");
            }
        };

        pool.submit(runA);
        pool.submit(runB);
        pool.shutdown();
    }
}

注意：RejectedExecutionHandler的实施取决于您！我只是建议将一个睡眠视为一种阻塞机制，但是你可以做一个更复杂的逻辑，因为询问线程池是否有空闲线程。如果没有，那么睡觉;如果是，则再次提交任务......

Answer 2

我找到了另一个选项，而不是@Carlitos Way提出的选项。它包括使用BlockingQueue.offer直接在队列中添加任务。我最初没有设法让它工作的唯一原因是我不得不发布这个问题，我不知道ThreadPoolExecutor的默认行为是在没有任何线程的情况下启动。线程将使用线程工厂延迟创建，并且可以删除/重新填充，具体取决于池的核心和最大大小以及同时提交的任务数。

由于线程创建是懒惰的，我阻止对offer的调用的尝试失败，因为如果没有人等待从队列中获取元素，SynchronousQueue.offer会立即退出。相反，SynchronousQueue.put阻塞，直到有人要求从队列中取一个项目，如果线程池为空，这将永远不会发生。

因此，解决方法是强制线程池使用ThreadPoolExecutor.prestartAllCoreThreads急切地创建核心线程。然后我的问题变得相当微不足道。我制作了一个真实用例的简化版本：

import java.util.Random;
import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.atomic.AtomicLong;

import static java.util.concurrent.TimeUnit.MILLISECONDS;
import static java.util.concurrent.TimeUnit.SECONDS;

public class SimplifiedBuildScheduler {
    private static final int MAX_POOL_SIZE = 10;

    private static final Random random = new Random();
    private static final AtomicLong nextTaskId = new AtomicLong(0);

    public static void main(String[] args) throws InterruptedException {
        SynchronousQueue<Runnable> queue = new SynchronousQueue<>();

        // this is a soft requirement in my system, not a real-time guarantee. See the complete semantics in my question.
        long maxBuildTimeInMillis = 50;
        // this timeout must be small compared to maxBuildTimeInMillis in order to accurately match the maximum build time
        long taskSubmissionTimeoutInMillis = 1;

        ThreadPoolExecutor pool = new ThreadPoolExecutor(MAX_POOL_SIZE, MAX_POOL_SIZE, 0, SECONDS, queue);
        pool.prestartAllCoreThreads();

        Runnable nextTask = makeTask(maxBuildTimeInMillis);

        long millisAtStart = System.currentTimeMillis();
        while (maxBuildTimeInMillis > System.currentTimeMillis() - millisAtStart) {
            boolean submitted = queue.offer(nextTask, taskSubmissionTimeoutInMillis, MILLISECONDS);
            if (submitted) {
                nextTask = makeTask(maxBuildTimeInMillis);
            } else {
                System.out.println("Task " + nextTaskId.get() + " was not submitted. " + "It will be rescheduled unless " +
                        "the max build time has expired");
            }
        }

        System.out.println("Max build time has expired. Stop submitting new tasks and running existing tasks to completion");

        pool.shutdown();
        pool.awaitTermination(9999999, SECONDS);
    }

    private static Runnable makeTask(long maxBuildTimeInMillis) {
        long sleepTimeInMillis = randomSleepTime(maxBuildTimeInMillis);
        long taskId = nextTaskId.getAndIncrement();
        return () -> {
            try {
                System.out.println("Task " + taskId + " sleeping for " + sleepTimeInMillis + " ms");
                Thread.sleep(sleepTimeInMillis);
                System.out.println("Task " + taskId + " completed !");
            } catch (InterruptedException ex) {
                throw new RuntimeException(ex);
            }
        };
    }

    private static int randomSleepTime(long maxBuildTimeInMillis) {
        // voluntarily make it possible that a task finishes after the max build time is expired
        return 1 + random.nextInt(2 * Math.toIntExact(maxBuildTimeInMillis));
    }
}

输出示例如下：

Task 1 was not submitted. It will be rescheduled unless the max build time has expired
Task 0 sleeping for 23 ms
Task 1 sleeping for 26 ms
Task 2 sleeping for 6 ms
Task 3 sleeping for 9 ms
Task 4 sleeping for 75 ms
Task 5 sleeping for 35 ms
Task 6 sleeping for 81 ms
Task 8 was not submitted. It will be rescheduled unless the max build time has expired
Task 8 was not submitted. It will be rescheduled unless the max build time has expired
Task 7 sleeping for 86 ms
Task 8 sleeping for 47 ms
Task 9 sleeping for 40 ms
Task 11 was not submitted. It will be rescheduled unless the max build time has expired
Task 2 completed !
Task 10 sleeping for 76 ms
Task 12 was not submitted. It will be rescheduled unless the max build time has expired
Task 3 completed !
Task 11 sleeping for 31 ms
Task 13 was not submitted. It will be rescheduled unless the max build time has expired
Task 13 was not submitted. It will be rescheduled unless the max build time has expired
Task 13 was not submitted. It will be rescheduled unless the max build time has expired
Task 13 was not submitted. It will be rescheduled unless the max build time has expired
Task 13 was not submitted. It will be rescheduled unless the max build time has expired
Task 13 was not submitted. It will be rescheduled unless the max build time has expired
Task 0 completed !
Task 12 sleeping for 7 ms
Task 14 was not submitted. It will be rescheduled unless the max build time has expired
Task 14 was not submitted. It will be rescheduled unless the max build time has expired
Task 1 completed !
Task 13 sleeping for 40 ms
Task 15 was not submitted. It will be rescheduled unless the max build time has expired
Task 12 completed !
Task 14 sleeping for 93 ms
Task 16 was not submitted. It will be rescheduled unless the max build time has expired
Task 16 was not submitted. It will be rescheduled unless the max build time has expired
Task 16 was not submitted. It will be rescheduled unless the max build time has expired
Task 5 completed !
Task 15 sleeping for 20 ms
Task 17 was not submitted. It will be rescheduled unless the max build time has expired
Task 17 was not submitted. It will be rescheduled unless the max build time has expired
Task 11 completed !
Task 16 sleeping for 27 ms
Task 18 was not submitted. It will be rescheduled unless the max build time has expired
Task 18 was not submitted. It will be rescheduled unless the max build time has expired
Task 9 completed !
Task 17 sleeping for 95 ms
Task 19 was not submitted. It will be rescheduled unless the max build time has expired
Max build time has expired. Stop submitting new tasks and running existing tasks to completion
Task 8 completed !
Task 15 completed !
Task 13 completed !
Task 16 completed !
Task 4 completed !
Task 6 completed !
Task 10 completed !
Task 7 completed !
Task 14 completed !
Task 17 completed !

例如，您会注意到任务19未重新安排，因为在调度程序第二次尝试将其提供给队列之前，最大构建时间已到期。您还可以看到在最长构建时间到期之前开始的所有正在进行的任务运行完成。

注意：正如我在代码中的注释所述，最长构建时间是 soft 要求，这意味着可能无法完全满足，并且我的解决方案确实允许在最长构建时间到期后提交任务。如果对offer的调用在最大构建时间到期之前启动并且之后完成，则会发生这种情况。为了减少它发生的几率，重要的是调用offer时使用的超时远小于最大构建时间。在实际系统中，线程池通常忙于没有空闲线程，因此这种竞争条件发生的概率非常小，并且当它确实发生时它对系统没有不良后果，因为最大构建时间是尽最大努力尝试满足整体运行时间，而不是精确而严格的约束。

使用ThreadPoolExecutor

2 个答案: