Question

我正在观察我无法用ExecutorService解释的行为。我有一个应用程序在内存中加载大约800人的配置文件。考虑以下

ExecutorService es = Executors.newFixedThreadPool(8); // 8 cores machine
Runnable enricherService = new Runnable() {

    @Override
    public void run() {
            doEnrichment(conn, tmpCachePerson);
    }
};

// tmpCachePerson is a ConcurrentLinkedQueue<Person>
while (tmpCachePerson.isEmpty() == false) {
    es.execute(enricherService);
}

es.shutdown();

try {
    while (!es.awaitTermination(24L, TimeUnit.HOURS)) {
        System.out.println("Waiting for termination");
    }
} catch (InterruptedException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
}

与1个线程的池相比，这块很慢。我在代码中放置了一些println，我可以看到每隔一段时间（~4s），所有线程停止，坐在那里最多16秒，然后重新开始，几乎像批处理，但在迭代之间休眠。完成需要50秒。然后我尝试了下面的实现：

Runnable enricher2Thread = new Runnable() {

            @Override
            public void run() {
                while (tmpCachePerson.isEmpty() == false) {
                    doEnrichment(conn, tmpCachePerson);
                }
            }
        };

        Thread t = new Thread(enricher2Thread);
        t.start();
        try {
            t.join();
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

另一方面，这篇文章非常快，只使用一个线程，从不停止在控制台中打印，并在3s内完成。

如果我用缓存池替换第一块中的固定池，则在生成800个线程后，任务在3秒内完成。如果我在固定池中放置800个线程，速度相同。任何人都明白为什么固定池会经常停顿，因此为什么它不会比1个线程更快。下面是我看到的8个线程的摘录。如果你看一下thread-1，它会暂停一个5s的简单getter。在日志的其他部分，整个任务需要大约250毫秒。

2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting executing SQL
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done executing SQL in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting adding
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting Getting Val
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done Getting Val in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting Getting Root
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done Getting Root in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-1 - Starting Getting Path
2016-11-15 15:54:04.212 - pool-1-thread-1 - Done Getting Path in 0 ms
2016-11-15 15:54:04.212 - pool-1-thread-6 - Starting
2016-11-15 15:54:04.212 - pool-1-thread-8 - Starting  <-------------- All threads stop
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting executing SQL
2016-11-15 15:54:09.533 - pool-1-thread-6 - Starting executing SQL
2016-11-15 15:54:09.533 - pool-1-thread-8 - Done executing SQL in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Starting Getting Full Path
2016-11-15 15:54:09.533 - pool-1-thread-6 - Done executing SQL in 5320 ms
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting adding
2016-11-15 15:54:09.533 - pool-1-thread-6 - Starting adding
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting Getting Val
2016-11-15 15:54:09.533 - pool-1-thread-6 - Starting Getting Val
2016-11-15 15:54:09.533 - pool-1-thread-8 - Done Getting Val in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting Getting Root
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done Getting Full Path in 5320 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Starting Adding Image
2016-11-15 15:54:09.533 - pool-1-thread-8 - Done Getting Root in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done Adding Image in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-8 - Starting Getting Path
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done adding in 5321 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Starting setting
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done setting in 0 ms
2016-11-15 15:54:09.533 - pool-1-thread-1 - Done in 5321 ms

知道如何改进此代码及其暂停的原因吗？如果有帮助，我可以发布doEnrichment（）的代码

编辑：这是：

private void doEnrichment(Connection conn, ConcurrentLinkedQueue<Person> tmpCachePerson) {
        Person person = tmpCachePerson.poll();

        if (person != null) {
            ImageCollection personImageCollection;
            String query = "SELECT epi.value, i.path FROM Image i "
                    + "INNER JOIN EntityImageRelationship eir ON eir.id_image = i.id "
                    + "INNER JOIN EntityType et ON eir.id_entity_type = et.id "
                    + "INNER JOIN EntityPrimaryImage epi ON epi.type_to_entity_uid = eir.type_to_entity_uid "
                    + "WHERE et.id = ? AND eir.id_entity_id = ? ORDER BY i.id ASC";

            String tagQuery = "SELECT id, value FROM Tag t INNER JOIN EntityTagRelationship etr ON etr.id_tag = t.id WHERE etr.id_entity = ? AND etr.id_entity_type = ?";

            try (PreparedStatement stmnt = conn.prepareStatement(query);
                    PreparedStatement tagStmnt = conn.prepareStatement(tagQuery)) {
                personImageCollection = getEntityImages(conn, stmnt, person.getId(), person.getMovieEntityType());
                person.setImageCollection(personImageCollection);
                person.getImageCollection().setPrimaryImageIcon();

                Set<String> tags = getEntityTags(conn, tagStmnt, person.getId(), person.getMovieEntityType()).keySet();
                person.setTags(tags);

                personCache.add(person);
            } catch (Exception e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
        }
    }

Answer 1

这可能不是原因，但你有竞争条件：

while (tmpCachePerson.isEmpty() == false) {
    es.execute(enricherService);
}

绝对不能保证上下文切换发生;即使它确实如此，它可能比你预期的要慢;在工人启动之前，该循环可能会运行一百万次。在你解决这个问题之前，不要再看了。这带来了大量的旋转和内存开销。

更好的模式是将轮询放在工作人员中：

Runnable enricherService = new Runnable() {
    @Override
    public void run() {
        while (!tmpCachePerson.isEmpty()) {
            doEnrichment(conn, tmpCachePerson);
            // TODO: error handling? Should a failure in doEnrichment kill the worker?
        }
    }
};

然后用

启动工人

for (int i = 0; i < 8; ++i) {
    es.execute(enricherService);
}

Java - 具有明显空闲线程的ExecutorService

1 个答案: