由于迫切需要，我今天刚刚学习了多线程编程。请帮我解决这个问题。
我有一个字符串处理任务，可以很好地划分为小子任务。
while (...){
...
// assign task for handler
Thread t = new Thread(new PCHandler(counter,pc));
t.start();
counter++;
}
问题是我需要大约500K线程来完成这项任务。我遇到了一个错误：
引起：java.lang.OutOfMemoryError：无法创建新的本机线程
我谷歌一段时间似乎JVM只允许我制作最大32K线程。有一些指令可以通过修改配置文件来扩展此限制。但我想避免修改用户的计算机。那么你能否给我一个如何在极限内明智地管理它们的建议？感谢。
答案 0 :(得分：21)
问题是我需要大约500K线程来完成这项任务。我遇到了[记忆错误]。
听起来你应该使用一个线程池，这样你就可以提交大量的作业，但只能在少量线程中运行它们。< / p>
// create a thread pool with 10 threads, this can be optimized to your hardware
ExecutorService threadPool = Executors.newFixedThreadPool(10);
// submit your handlers to the thread-pool
for (PCHandler handler : handlersToDo) {
threadPool.submit(handler);
}
// once we have submitted all jobs to the thread pool, it should be shutdown
threadPool.shutdown();
...
如果这不起作用，那么我想知道有关实际需要500k并发运行线程的系统的更多细节。您可以通过一些内存设置调整和增加盒子上的核心内存来实现这一点，但我怀疑重新构建您的应用程序是有序的。
正如@Peter在评论中提到的，为了优化池中的线程数，您可以获得可用处理器的数量和其他系统规格来解决这个问题。但这在很大程度上取决于PCHandler类的CPU密集程度。它做的IO越多，可以利用的并发性就越多。可能会使用传递给newFixedThreadPool(...)方法的不同值进行一些测试运行，以确定那里的最佳设置。
答案 1 :(得分：1)
除非是16核或更高的核心计算机，否则绝对不是一个由单个应用程序在一台计算机上管理这么多线程的好选择。
请考虑一些因素，例如您的工作是I / O密集型还是CPU密集型，并做出适当的选择。阅读here和here
我通常使用
int maxThreadCount = Runtime.getRuntime().availableProcessors();
ExecutorService executor =
new ThreadPoolExecutor(
0, maxThreadCount - 1,
1, TimeUnit.SECONDS,
new LinkedBlockingDeque<>(maxThreadCount * 2),
Executors.defaultThreadFactory(),
new ThreadPoolExecutor.CallerRunsPolicy());
现在通过添加任务来进行处理，并等待一切完成：
while (moreTaskstoDo) {
Callable c =...
executor.submit(c);
}
executor.shutdown();
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
现在有了Java 8+，您可以考虑更高效地完成它。
我做了一个小的基准测试。以下代码受article的启发，您可以阅读有关Java 8 Handbook
的更多信息考虑此求和功能。
//approach 1: old school
private static void findingTotalOldSchool() {
long total = 0;
long start = System.nanoTime();
for (long i = 1; i < LIMIT; i++) {
total = total + (i * FACTOR);
}
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
public static Range range(int max) {
return new Range(max);
}
// Approach 2: custom iterator
private static void findingTotalCustomIterator() {
long total = 0;
long start = System.nanoTime();
for (long i : range(LIMIT)) {
total = total + i * FACTOR;
}
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
// Approach 3: using streams
private static void findingTotalStream() {
long start = System.nanoTime();
long total = 0;
total = LongStream.range(1, LIMIT)
.map(t -> t * FACTOR)
.sum();
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
// Approach 4: using parallel streams
private static void findingTotalParallelStream() {
long start = System.nanoTime();
long total = 0;
total = LongStream.range(1, LIMIT)
.parallel()
.map(t -> t * FACTOR)
.sum();
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
// Approach 5: Using Completable Futures alone
private static void findingTotalCFS() {
long start = System.nanoTime();
List<CompletableFuture<Long>> futures =
LongStream.range(1, LIMIT).boxed()
.map(t -> CompletableFuture.supplyAsync(() -> t * FACTOR ))
.collect(Collectors.toList());
//Code here --- could run ahead hence joining on futures
long total = futures.stream().map(CompletableFuture::join).mapToLong(t->t).sum();
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Futures used: "+futures.size());
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
// Approach 6: Using Completable Futures managed by Executor Service
private static void findingTotalCFSE() {
long start = System.nanoTime();
ExecutorService executor = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() + 1);
List<CompletableFuture<Long>> futures =
LongStream.range(1, LIMIT).boxed()
.map(t -> CompletableFuture.supplyAsync(() -> {
return t * FACTOR;
}, executor))
.collect(Collectors.toList());
long total = futures.stream().map(CompletableFuture::join).mapToLong(t->t).sum();
executor.shutdownNow();
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Futures used: "+futures.size());
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
// Approach 7: Using Executor service alone
private static void findingTotalES() {
long start = System.nanoTime();
ExecutorService executorService = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors() + 1);
long total = LongStream.
range(1, LIMIT)
.boxed()
.map((i)->executorService.submit(new Operation(i, FACTOR)))
.map((Future<Long> future)-> {
try {
return future.get();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}catch (ExecutionException e) {
// Extract the actual exception from its wrapper
Throwable t = e.getCause();
}
return 0;
})
.mapToLong(t->t.longValue())
.sum();
executorService.shutdown();
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Duration: "+duration);
System.out.println("Total: "+total);
}
class Operation implements Callable<Long> {
long i; int j;
Operation(long i, int j) { this.i = i; this.j = j; }
@Override
public Long call() {
return i * j;
}
}
class Range implements Iterable<Integer> {
private int limit;
public Range(int limit) {
this.limit = limit;
}
@Override
public Iterator<Integer> iterator() {
final int max = limit;
return new Iterator<Integer>() {
private int current = 0;
@Override
public boolean hasNext() {
return current < max;
}
@Override
public Integer next() {
if (hasNext()) {
return current++;
} else {
throw new NoSuchElementException("Range reached the end");
}
}
@Override
public void remove() {
throw new UnsupportedOperationException("Can't remove values from a Range");
}
};
}
}
我们用2组数据进行了测试。每个测试应单独运行，而不应作为单个整体运行的一部分（随着JVM优化，结果可能会有所不同）。
//first run
final static int FACTOR = 1;
final static int LIMIT = 10000;
//second run
final static int FACTOR = 9876;
final static int LIMIT = 1000000;
System.out.println("-----Traditional Loop-----");
findingTotalOldSchool();
// 0 ms
// 4 ms
System.out.println("-----Custom Iterator----");
findingTotalCustomIterator();
// 1 ms
// 15 ms
System.out.println("-----Streams-----");
findingTotalStream();
// 38 ms
// 33 ms
System.out.println("-----Parallel Streams-----");
findingTotalParallelStream();
// 29 ms
// 64 ms
System.out.println("-----Completable Futures with Streams-----");
findingTotalCFS();
// 77 ms
// 635 ms
System.out.println("-----Executor Service with Streams-----");
findingTotalES();
// 323 ms
// 12632 ms
System.out.println("-----Completable Futures with Executor Service with Streams-----");
findingTotalCFSE();
// 77 ms
// 844 ms
观察：