Question

有没有办法通过一个庞大的数据库并同时应用一些工作条目？我尝试使用ExecutorService，但我们必须shutdown（）才能知道池大小......

所以我最好的解决方案是：

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

public class TestCode
{
private static List<String> getIds(int dbOffset, int nbOfArticlesPerRequest) 
{
    return Arrays.asList("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29");
}

public static void main(String args[]) throws Exception
{
    int dbOffset = 0;
    int nbOfArticlesPerRequest = 100;
    int MYTHREADS = 10;
    int loopIndex = 0;
    boolean bContinue=true;
    Runnable worker;



    while(bContinue) // in this loop we'll constantly fill the pool list
    {
        loopIndex++;
        ExecutorService executor = Executors.newFixedThreadPool(MYTHREADS); // NOT IDEAL, BUT EXECUTORSERVICE CANNOT BE REUSED ONCE SHUTDOWN...

        List<String> ids = getIds(dbOffset, nbOfArticlesPerRequest ); // getIds(offset, rows_number)
        for(String id: ids) {
            worker = new MyRunnable(id);
            executor.execute(worker);
        }

        executor.shutdown();
        while (!executor.isTerminated()) {
            System.out.println("Pool size is now " + ((ThreadPoolExecutor) executor).getActiveCount()+
                    " - queue size: "+ ((ThreadPoolExecutor) executor).getQueue().size()
            );
            TimeUnit.MILLISECONDS.sleep(500);
        }

        if(loopIndex>=3) {
            System.out.println("\nEnd the loop #"+loopIndex+" ===> STOOOP!\n");
            bContinue = false;
        }
        dbOffset+=nbOfArticlesPerRequest;
    }
}



public static class MyRunnable implements Runnable {

    private final String id;

    MyRunnable(String id) {
        this.id = id;
    }

        @Override
        public void run()
        {
            System.out.println("Thread '"+id+"' started");
            try {
                TimeUnit.MILLISECONDS.sleep(2000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            System.out.println("Thread '"+id+"' stopped");
        }
    }
}

这工作正常，但缺点是在循环的每一端我都需要等待最后一个线程完成。

例如：当只有3个线程正在运行时......

我为了解决这个问题做了以下工作，但是“安全”/正确吗？

BTW：有没有办法知道队列中有多少个任务/线程？

    int dbOffset = 0;
    int nbOfArticlesPerRequest = 5; //100;
    int MYTHREADS = 2;
    int loopIndex = 0;

    ExecutorService executor = Executors.newFixedThreadPool(MYTHREADS); // **HERE IT WOULD BE A GLOBAL VARIABLE**
       while(bContinue) // in this loop we'll constantly fill the pool list
        {
            loopIndex++;

            List<String> ids = getIds(dbOffset, nbOfArticlesPerRequest ); // getIds(offset, rows_number)
             for(String id: ids) {
                    worker = new MyRunnable(id);
                    executor.execute(worker);
             }

            while (!executor.isTerminated() && ((ThreadPoolExecutor) executor).getActiveCount() >= MYTHREADS) {
                System.out.println("Pool size is now " + ((ThreadPoolExecutor) executor).getActiveCount()+
                        " - queue size: "+ ((ThreadPoolExecutor) executor).getQueue().size()
                );
                TimeUnit.MILLISECONDS.sleep(500);
            }

            if(loopIndex>=3) {
                System.out.println("\nEnd the loop #"+loopIndex+" ===> STOOOP!\n");
                bContinue = false;
            }
            dbOffset+=nbOfArticlesPerRequest;
        }

    executor.shutdown();
    // Wait until all threads are finish
    while (!executor.isTerminated()) {
        System.out.println("Pool size is now " + ((ThreadPoolExecutor) executor).getActiveCount()+
                " - queue size: "+ ((ThreadPoolExecutor) executor).getQueue().size()
        );
        TimeUnit.MILLISECONDS.sleep(500);
    }

修改

我尝试启动1或10百万个任务，所以（我假设）我不能将它们全部放入队列......这就是为什么我使用全局执行程序以便能够始终拥有一些线程的原因队列（因为我无法关闭执行程序，否则它不再可用）。

最新代码版本：

    int dbOffset = 0;
    int nbOfArticlesPerRequest = 5; //100;
    int MYTHREADS = 2;
    int loopIndex = 0;

    ThreadPoolExecutor executorPool = new ThreadPoolExecutor(MYCORES, MYCORES, 0L,TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>()); // **HERE IT WOULD BE A GLOBAL VARIABLE**
       while(bContinue) // in this loop we'll constantly fill the pool list
        {
            loopIndex++;

            List<String> ids = getIds(dbOffset, nbOfArticlesPerRequest ); // getIds(offset, rows_number)
             for(String id: ids) {
                    worker = new MyRunnable(id);
                    executorPool.execute(worker);
             }

            while (executorPool.getActiveCount() >= MYTHREADS  || executorPool.getQueue().size()> Math.max(1, MYTHREADS -2)) 
            {
                System.out.println("Pool size is now " + executorPool.getActiveCount()+
                                        " - queue size: "+ executorPool.getQueue().size()
                );

                if(executorPool.getQueue().size() <= Math.max(1, MYCORES-2)) {
                    System.out.println("Less than "+Math.max(1, MYCORES-2)+" threads in queue ---> fill the queue");
                    break;
                }

                TimeUnit.MILLISECONDS.sleep(2000);
            }

            if(loopIndex>=3) {
                System.out.println("\nEnd the loop #"+loopIndex+" ===> STOOOP!\n");
                bContinue = false;
            }
            dbOffset+=nbOfArticlesPerRequest;
        }

    executorPool.shutdown();
    // Wait until all threads are finish
    while (!executorPool.isTerminated()) {
        System.out.println("Pool size is now " + executorPool.getActiveCount()+
                " - queue size: "+ executorPool.getQueue().size()
        );
        TimeUnit.MILLISECONDS.sleep(500);
    }

提前致谢

Answer 1

<强>更新

现在我很清楚，你主要担心的是你不能同时提交1000万个任务。

不要害怕，你可以将所有这些都提交给遗嘱执行人。并行运行的实际任务量受底层线程池大小的限制。也就是说，如果池大小为2，那么当时只执行两个任务，其余任务都在队列中等待自由线程。

默认情况下，Executors.newFixedThreadPool()会创建一个队列大小为Integer.MAX_VALUE的执行程序，因此您可以在那里完成数百万个任务。

您可以使用返回ExecutorService.submit()的{{1}}方法。然后，您可以检查未来任务的状态（即使用Future，isDone()方法。）

执行程序通常是您不希望显式关闭并且在整个应用程序生命周期中存在的内容。使用这种方法，您不需要关闭以了解待处理的任务数量。

isCancelled()

此外，请注意，任务和线程不是可互换的术语。在您的情况下，执行程序具有固定数量的线程。您可以提交更多的任务，但其余任务将位于执行程序的队列中，直到有一个免费线程来运行任务。

Answer 2

ExecuterService允许您调用可以并行运行的任务列表，并在结果可用时返回结果。

在您使用的代码中

worker = new MyRunnable(id);
executor.execute(worker);

而不是Runnable，在这个用例中使用Callable会更好，那么你可以在单个api而不是for循环中提交Callables列表以供执行。

List<Callable> workers = new ArrayList<>();
workers.add(new MyCallable(id)) // this is just for example
workers.add(new MyCallable(id))
workers.add(new MyCallable(id))

List<Future<Boolean>> futures = executor.invokeAll(workers); // this will execute all worker tasks parallely and return you future object list using which you can see whether worker thread is completed or not and also the what is the return value.

请注意，Future对象的get方法是阻塞调用

Answer 3

您不需要知道线程池大小来检查ExecutorService中任务的完成情况。您可以在提交任务后删除代码。

选项1：

用Executors中的newWorkStealingPool替换ThreadPoolExecutor。

使用所有可用处理器作为目标并行级别创建工作窃取线程池。

它将允许更好地利用ExecutorService中的线程。
```
ExecutorService executor = Executors.newWorkStealingPool();
```
使用invokeAll

选项2：（如果您事先知道任务数量，则非常有用）

使用CountDownLatch并将计数器初始化为要提交的任务数。

进一步参考：

wait until all threads finish their work in java

How to properly shutdown java ExecutorService

Java - ExecutorService的最大大小为

3 个答案: