Question

我们当前正在使用以下log4j jars

log4j-core-2.6.2，log4j-flume-ng-2.6.2

<Flume name="aggregatorApp" compress="false" type="Avro"
ignoreExceptions="true" batchSize="10" blocking="false" >
<Agent host="${flumeHostPrimary}" port="${flumePortPrimary}" />
<Agent host="${flumeHostSecondary}" port="${flumePortSecondary}" />
<FlumeEventFactory logType="APPLICATION"/>
<PatternLayout header="app" pattern="[%-5p] %d %c %X{correlationId} - %m" />

<AsyncLogger name="com.xyz.abc" level="debug"
additivity="false" blocking="false">
<AppenderRef ref="IDRESTSERVICELOG" />
<AppenderRef ref="aggregatorApp" level="INFO" />
<AppenderRef ref="CONSOLE" level="INFO"/>
</AsyncLogger>

启动水槽代理并且正在运行的应用程序运行顺利，并且日志事件已成功发布到水槽代理
如果水槽代理关闭了一段时间，则应用程序线程将被以下异常卡住

Exception in thread "elasticsearch[_client_][generic][T#3]" java.lang.OutOfMemoryError: GC overhead limit exceeded
2019-02-01 07:43:23,618 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] WARN  org.apache.catalina.valves.StuckThreadDetectionValve- Thread "http-apr-8080-exec-7" (id=167) has been active for 20,392 milliseconds (since 2/1/19 7:42 AM) to serve the same request for htpps://xyz//directories/v1.0/search and may be stuck (configured threshold for this StuckThreadDetectionValve is 20 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.
2019-02-01 07:46:18,619 [ContainerBackgroundProcessor[StandardEngine[Catalina]]] ERROR org.apache.catalina.core.ContainerBase- Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[Catalina]]
java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-apr-8080-exec-42" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "http-apr-8080-exec-50" Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" Exception in thread "I/O dispatcher 23" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"
Exception in thread "http-apr-8080-exec-38" Exception in thread "http-apr-8080-exec-46" Exception in thread "http-apr-8080-exec-53" Exception in thread "I/O dispatcher 17" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-apr-8080-exec-31"

在高负载（每个请求1MB日志记录）期间发生上述异常。注意：我们正在进行负载测试

由于应用程序记录繁重，但Flume Appender无法以相同的速度消耗。我们怀疑Log4j环形缓冲区已满，这会阻塞应用程序线程并创建内存不足异常。

考虑到Log4J附加程序存在某些问题，我们尝试了一些操作，

-Dlog4j2.AsyncQueueFullPolicy =丢弃-Dlog4j2.enable.threadlocals = true -Dlog4j2.enable.direct.encoders = true

但以上系统属性无用。我们错过了什么吗？

Answer 1

我同意AsyncLogger的RingBuffer导致您的内存不足。环形缓冲区的默认大小为256K条目。 JVM的堆大小是多少？

另一种方法是使用FlumeAppender的嵌入式或持久版本。这些将在转发到Flume代理之前将事件缓存在本地文件通道中。这将允许您通过调整可用磁盘空间来控制服务器可以容忍多长时间。您仍然可以利用异步记录器的优势，但是可以避免中断。

Flume Appender问题：在大量日志记录期间Flume Agent关闭时，应用程序线程被卡住

1 个答案: