如何在ThreadPoolExecutor程序中解决java.lang.outOfMemoryError?

时间:2014-08-17 04:02:52

标签: java multithreading memory memory-management threadpool

我正在运行一个ThreadPoolExecutor程序,它创建1000个url并发送给执行程序服务。

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:673)
    at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
    at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
    at MyProgramName.main(MyProgramName.java:175)
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:20 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
2014-08-16 20:46:21.341 java[5100:d07] java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:673)
    at sun.awt.AWTAutoShutdown.activateBlockerThread(AWTAutoShutdown.java:312)
    at sun.awt.AWTAutoShutdown.setToolkitBusy(AWTAutoShutdown.java:232)
    at sun.awt.AWTAutoShutdown.notifyToolkitThreadBusy(AWTAutoShutdown.java:118)
2014-08-16 20:46:21.342 java[5100:d07] (
    0   CoreFoundation                      0x00007fff9127825c __exceptionPreprocess + 172
    1   libobjc.A.dylib                     0x00007fff9031ce75 objc_exception_throw + 43
    2   CoreFoundation                      0x00007fff91277e09 -[NSException raise] + 9
    3   JavaNativeFoundation                0x000000010c754c3f JNFCallStaticVoidMethod + 213
    4   libawt.jnilib                       0x0000000115fb8151 setBusy + 53
    5   CoreFoundation                      0x00007fff911a8d67 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
    6   CoreFoundation                      0x00007fff911a8cd7 __CFRunLoopDoObservers + 391
    7   CoreFoundation                      0x00007fff9119a608 __CFRunLoopRun + 1368
    8   CoreFoundation                      0x00007fff91199e75 CFRunLoopRunSpecific + 309
    9   HIToolbox                           0x00007fff8fdd9a0d RunCurrentEventLoopInMode + 226
    10  HIToolbox                           0x00007fff8fdd97b7 ReceiveNextEventCommon + 479
    11  HIToolbox                           0x00007fff8fdd95bc _BlockUntilNextEventMatchingListInModeWithFilter + 65
    12  AppKit                              0x00007fff8ac5d24e _DPSNextEvent + 1434
    13  AppKit                              0x00007fff8ac5c89b -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 122
    14  libawt.jnilib                       0x0000000115fb987f -[NSApplicationAWT nextEventMatchingMask:untilDate:inMode:dequeue:] + 124
    15  AppKit                              0x00007fff8ac5099c -[NSApplication run] + 553
    16  libawt.jnilib                       0x0000000115fb7f30 +[AWTStarter startAWT:] + 1495
    17  libawt.jnilib                       0x0000000115fb78aa -[CPerformer perform] + 93
    18  Foundation                          0x00007fff8c64313e __NSThreadPerformPerform + 229
    19  CoreFoundation                      0x00007fff911a95b1 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
    20  CoreFoundation                      0x00007fff9119ac62 __CFRunLoopDoSources0 + 242
    21  CoreFoundation                      0x00007fff9119a3ef __CFRunLoopRun + 831
    22  CoreFoundation                      0x00007fff91199e75 CFRunLoopRunSpecific + 309
    23  java                                0x000000010be86843 java + 18499
    24  java                                0x000000010be8629a java + 17050
    25  java                                0x000000010be83a98 java + 6808
    26  ???                                 0x0000000000000005 0x0 + 5
)
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->MyURLURLURLURLRURL Connection reset
Aug 16, 2014 8:46:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->MyURLURLURLURLRURL

我已经在互联网和stackoverflow上做了一些研究。一些答案说我需要减少堆栈大小。有人说我需要使用ulimit -a来增加ulimit。我完全糊涂了。我对这些答案感到困惑,因为它们并不特定于ThreadExecutor。请帮帮我

我在Mac OS,i7四核处理器,16 GB RAM,1 Tb HD中运行代码

我的计划代码在这个问题Why does the following executor service java Thread program doesn't shut down?

2 个答案:

答案 0 :(得分:4)

真正的问题是你正在尝试使用过多的堆栈内存来创建太多的线程。

您可以减少线程堆栈的默认大小(如建议的那样),但这不是一个完整的解决方案。 (当你决定在更多网址上运行你的程序时,你会再次遇到同样的问题。最终你会遇到“墙”,你无法进一步减少堆栈大小;例如,因为你得到{{1例外。)

更好的解决方案是不要创建这么多线程。您已将程序配置为使用无限线程池。这坦率地说是荒谬的。改变这个:

StackOverflowError

更现实:

ThreadPoolExecutor executor = 
      new ThreadPoolExecutor(Integer.MAX_VALUE, Integer.MAX_VALUE, 
                             20, TimeUnit.MILLISECONDS, blockingQueue);

超过某一点,添加额外的线程不会使应用程序更快。实际上,它通常会开始使应用程序变慢,有时灾难性更慢。

答案 1 :(得分:0)

当JVM从操作系统询问新线程时,您将面临“java.lang.OutOfMemoryError:无法创建新的本机线程”。只要底层操作系统无法分配新的本机线程,就会抛出此OutOfMemoryError。本机线程的确切限制是非常依赖于平台的,因此我们建议通过运行一个小测试找出您的限制,以找出特定于平台的线程限制。但是,通常情况下,导致 java.lang.OutOfMemoryError:无法创建新的本机线程的情况会经历以下阶段:

  1. JVM内部运行的应用程序请求新的Java线程
  2. JVM本机代码代理创建操作系统的新本机线程的请求
  3. OS尝试创建一个新的本机线程,该线程需要将内存分配给线程
  4. 操作系统将拒绝本机内存分配,因为32位Java进程大小耗尽了其内存地址空间 - 例如(2-4)GB流程大小限制已被命中 - 或者OS的虚拟内存已完全耗尽
  5. java.lang.OutOfMemoryError:无法创建新的本机线程错误被抛出
  6. 在您的情况下,克服该问题的最佳方法是遵循建议的限制线程创建的路径。