当目标在空闲超时后断开tcp连接时，使用log4j2套接字appender丢失初始日志事件

时间：2017-05-15 16:06:51

标签： java sockets logging log4j2 tcp-ip

在我们的应用程序中使用log4j2套接字appender将日志发送到日志目标时，我们正面临着间歇性的日志丢失。详细信息如下;

我们的环境：

  1. 在我们的应用程序中，我们使用log4j2（2.7版本jar）框架和SocketAppender（socket appender下的async appender）通过radware（负载均衡器）将日志发送到Splunk（企业日志记录工具）。
  2. 因此我们的应用程序使用SocketAppender通过TCPIP套接字连接到Radware; Radware用于在4个目标Splunk服务器之间平衡负载。
  3. 如果连接空闲一分钟（可在radware配置并设置为1分钟），Radware Server配置为超时

  4. log4j2配置文件的配置如下：

    <?xml version="1.0" encoding="UTF-8"?>
<Configuration status="trace" monitorInterval="5">
    <Appenders>
        <Socket name="socket" host="10.52.214.26" port="1514" reconnectDelayMillis="30000" immediateFail="false" bufferedIo="true" bufferSize="204800" protocol="TCP" immediateFlush="false">
            <PatternLayout></PatternLayout>
        </Socket>
        <Async name="SplunkAsync" bufferSize="204800">
            <AppenderRef ref="socket"/>
        </Async>
    </Appenders>
<Loggers>
    <Root level="warn">
        <AppenderRef ref="SplunkAsync"/>
    </Root>
    <Logger name="SplunkInfoLogger" level="info" additivity="false">
        <AppenderRef ref="SplunkAsync"/>
    </Logger>
    <Logger name="SplunkDebugLogger" level="debug" additivity="false">
        <AppenderRef ref="SplunkAsync"/>
    </Logger>
    <Logger name="SplunkWarnLogger" level="warn" additivity="false">
        <AppenderRef ref="SplunkAsync"/>
    </Logger>
</Loggers>

    5. 遇到问题的过程/步骤：

    1. Splunk服务器，radware服务器和我们的应用程序服务器都已启动，应用程序服务器成功地将日志发送到radware服务器。
    2. 由于我们没有进行任何操作，因此在一分钟内（我们设置为1分钟）没有日志进入无线电设备，并且radware破坏了它与应用服务器之间的连接。
    3. 我们做了一个操作，以便应用服务器写一个日志。但是没有任何反应，这个日志根本没有进入radware和splunk服务器。
    4. 我们进行了第二次操作，以便应用程序服务器编写第二个日志。这次日志也没有进入radware和splunk。
    5. 从第一个失败的日志（即radware断开连接后触发的日志）计数30秒（在log4j2配置文件中reconnectDelayMillis="30000"）后，我们进行了第三次操作，以便应用服务器写入第三个日志。这次SocketAppender运行良好，这个日志进入了radware和splunk服务器。这主要是因为应用服务器能够在30秒后以reconnectDelayMillis="30000"
      6. 重新建立与Radware的连接

      我想知道为什么在Radware连接变为空闲后发送的初始日志丢失了。理想情况下，log4j2创建的tcp连接应该在连接断开时将数据保留在缓冲区中，并且应该在30秒后建立重新连接时从缓冲区发送数据（reconnectDelayMillis =“30000”）

      我们尝试设置参数如immediateFail =“false”bufferedIo =“true”bufferSize =“204800”，这样当radware超时tcp连接时数据可以存储在缓冲区中，并且可以在以后重新连接到radware时发送，不幸的是，这没有发生。

      我们是否缺少任何配置，因为我们正面临这种行为，或者它是否已知缺陷。

      其他人也观察到类似的问题，并在apache jira中提出（https://issues.apache.org/jira/browse/LOG4J2-1311）

      -------------------
Console Log
---------------------
Configuration(C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml), connectTimeoutMillis="null", host="10.52.214.26", immediateFail="false", port="1514", protocol="TCP", reconnectDelayMillis="30000", SslConfiguration=null, bufferedIo="true", bufferSize="204800", immediateFlush="false", ignoreExceptions="null", PatternLayout(%m%n), name="socket", Filter=null)
2017-05-15 17:27:15.053     53 2017-05-15 17:27:15,052 Thread-39 DEBUG Starting TcpSocketManager TCP:10.52.214.26:1514
2017-05-15 17:27:15.054     53 2017-05-15 17:27:15,054 Thread-39 DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2017-05-15 17:27:15.066     53 2017-05-15 17:27:15,066 Thread-39 DEBUG createAppenderRef(ref="socket", level="null", Filter=null)
2017-05-15 17:27:15.070     53 2017-05-15 17:27:15,070 Thread-39 DEBUG Building Plugin[name=appender, class=org.apache.logging.log4j.core.appender.AsyncAppender].
2017-05-15 17:27:15.081     53 2017-05-15 17:27:15,081 Thread-39 DEBUG AsyncAppender$Builder(={socket}, errorRef="null", blocking="null", shutdownTimeout="null", bufferSize="204800", name="SplunkAsync", includeLocation="null", Filter=null, Configuration(C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml), ignoreExceptions="null", BlockingQueueFactory=null)
2017-05-15 17:27:15.084     53 2017-05-15 17:27:15,083 Thread-39 DEBUG Building Plugin[name=appenders, class=org.apache.logging.log4j.core.config.AppendersPlugin].
2017-05-15 17:27:15.086     53 2017-05-15 17:27:15,085 Thread-39 DEBUG createAppenders(={socket, SplunkAsync})
2017-05-15 17:27:15.087     53 2017-05-15 17:27:15,087 Thread-39 DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2017-05-15 17:27:15.089     53 2017-05-15 17:27:15,089 Thread-39 DEBUG createAppenderRef(ref="SplunkAsync", level="null", Filter=null)
2017-05-15 17:27:15.091     53 2017-05-15 17:27:15,090 Thread-39 DEBUG Building Plugin[name=root, class=org.apache.logging.log4j.core.config.LoggerConfig$RootLogger].
2017-05-15 17:27:15.094     53 2017-05-15 17:27:15,094 Thread-39 DEBUG createLogger(additivity="null", level="WARN", includeLocation="null", ={SplunkAsync}, ={}, Configuration(C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml), Filter=null)
2017-05-15 17:27:15.115     53 2017-05-15 17:27:15,113 Thread-39 DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2017-05-15 17:27:15.120     53 2017-05-15 17:27:15,120 Thread-39 DEBUG createAppenderRef(ref="SplunkAsync", level="null", Filter=null)
2017-05-15 17:27:15.123     53 2017-05-15 17:27:15,123 Thread-39 DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2017-05-15 17:27:15.128     53 2017-05-15 17:27:15,127 Thread-39 DEBUG createLogger(additivity="false", level="INFO", name="SplunkInfoLogger", includeLocation="null", ={SplunkAsync}, ={}, Configuration(C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml), Filter=null)
2017-05-15 17:27:15.130     53 2017-05-15 17:27:15,130 Thread-39 DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2017-05-15 17:27:15.133     53 2017-05-15 17:27:15,132 Thread-39 DEBUG createAppenderRef(ref="SplunkAsync", level="null", Filter=null)
2017-05-15 17:27:15.134     53 2017-05-15 17:27:15,134 Thread-39 DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2017-05-15 17:27:15.147     53 2017-05-15 17:27:15,146 Thread-39 DEBUG createLogger(additivity="false", level="DEBUG", name="SplunkDebugLogger", includeLocation="null", ={SplunkAsync}, ={}, Configuration(C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml), Filter=null)
2017-05-15 17:27:15.148     53 2017-05-15 17:27:15,148 Thread-39 DEBUG Building Plugin[name=AppenderRef, class=org.apache.logging.log4j.core.config.AppenderRef].
2017-05-15 17:27:15.151     53 2017-05-15 17:27:15,150 Thread-39 DEBUG createAppenderRef(ref="SplunkAsync", level="null", Filter=null)
2017-05-15 17:27:15.152     53 2017-05-15 17:27:15,152 Thread-39 DEBUG Building Plugin[name=logger, class=org.apache.logging.log4j.core.config.LoggerConfig].
2017-05-15 17:27:15.156     53 2017-05-15 17:27:15,156 Thread-39 DEBUG createLogger(additivity="false", level="WARN", name="SplunkWarnLogger", includeLocation="null", ={SplunkAsync}, ={}, Configuration(C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml), Filter=null)
2017-05-15 17:27:15.158     53 2017-05-15 17:27:15,157 Thread-39 DEBUG Building Plugin[name=loggers, class=org.apache.logging.log4j.core.config.LoggersPlugin].
2017-05-15 17:27:15.165     53 2017-05-15 17:27:15,165 Thread-39 DEBUG createLoggers(={root, SplunkInfoLogger, SplunkDebugLogger, SplunkWarnLogger})
2017-05-15 17:27:15.167     53 2017-05-15 17:27:15,167 Thread-39 DEBUG Configuration XmlConfiguration[location=C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml] initialized
2017-05-15 17:27:15.169     53 2017-05-15 17:27:15,168 Thread-39 DEBUG Starting configuration XmlConfiguration[location=C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml]
2017-05-15 17:27:15.178     53 2017-05-15 17:27:15,178 Thread-39 DEBUG Started configuration XmlConfiguration[location=C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml] OK.
2017-05-15 17:27:15.180     53 2017-05-15 17:27:15,180 Thread-39 TRACE Stopping org.apache.logging.log4j.core.config.DefaultConfiguration@ce510e6c...
2017-05-15 17:27:15.181     53 2017-05-15 17:27:15,181 Thread-39 TRACE DefaultConfiguration notified 1 ReliabilityStrategies that config will be stopped.
2017-05-15 17:27:15.184     53 2017-05-15 17:27:15,184 Thread-39 TRACE DefaultConfiguration stopping root LoggerConfig.
2017-05-15 17:27:15.185     53 2017-05-15 17:27:15,185 Thread-39 TRACE DefaultConfiguration notifying ReliabilityStrategies that appenders will be stopped.
2017-05-15 17:27:15.187     53 2017-05-15 17:27:15,186 Thread-39 TRACE DefaultConfiguration stopping remaining Appenders.
2017-05-15 17:27:15.188     53 2017-05-15 17:27:15,187 Thread-39 DEBUG Shutting down OutputStreamManager SYSTEM_OUT.false.false-1
2017-05-15 17:27:15.189     53 2017-05-15 17:27:15,189 Thread-39 DEBUG Shut down OutputStreamManager SYSTEM_OUT.false.false-1, all resources released: true
2017-05-15 17:27:15.190     53 2017-05-15 17:27:15,190 Thread-39 TRACE DefaultConfiguration stopped 1 remaining Appenders.
2017-05-15 17:27:15.191     53 2017-05-15 17:27:15,191 Thread-39 TRACE DefaultConfiguration cleaning Appenders from 1 LoggerConfigs.
2017-05-15 17:27:15.193     53 2017-05-15 17:27:15,193 Thread-39 DEBUG Stopped org.apache.logging.log4j.core.config.DefaultConfiguration@ce510e6c OK
2017-05-15 17:27:15.198     53 2017-05-15 17:27:15,197 Thread-39 TRACE Reregistering MBeans after reconfigure. Selector=org.apache.logging.log4j.core.selector.ClassLoaderContextSelector@7a7e9cb6
2017-05-15 17:27:15.199     53 2017-05-15 17:27:15,199 Thread-39 TRACE Reregistering context (1/1): 'c1edad85' org.apache.logging.log4j.core.LoggerContext@c1b7efc0
2017-05-15 17:27:15.200     53 2017-05-15 17:27:15,200 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85'
2017-05-15 17:27:15.202     53 2017-05-15 17:27:15,202 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=StatusLogger'
2017-05-15 17:27:15.203     53 2017-05-15 17:27:15,203 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=ContextSelector'
2017-05-15 17:27:15.205     53 2017-05-15 17:27:15,205 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=Loggers,name=*'
2017-05-15 17:27:15.213     53 2017-05-15 17:27:15,213 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=Appenders,name=*'
2017-05-15 17:27:15.215     53 2017-05-15 17:27:15,214 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=AsyncAppenders,name=*'
2017-05-15 17:27:15.216     53 2017-05-15 17:27:15,216 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=AsyncLoggerRingBuffer'
2017-05-15 17:27:15.218     53 2017-05-15 17:27:15,217 Thread-39 TRACE Unregistering but no MBeans found matching 'org.apache.logging.log4j2:type=c1edad85,component=Loggers,name=*,subtype=RingBuffer'
2017-05-15 17:27:15.220     53 2017-05-15 17:27:15,220 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85
2017-05-15 17:27:15.225     53 2017-05-15 17:27:15,225 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=StatusLogger
2017-05-15 17:27:15.234     53 2017-05-15 17:27:15,234 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=ContextSelector
2017-05-15 17:27:15.238     53 2017-05-15 17:27:15,237 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=Loggers,name=SplunkWarnLogger
2017-05-15 17:27:15.240     53 2017-05-15 17:27:15,240 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=Loggers,name=
2017-05-15 17:27:15.242     53 2017-05-15 17:27:15,242 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=Loggers,name=SplunkDebugLogger
2017-05-15 17:27:15.243     53 2017-05-15 17:27:15,243 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=Loggers,name=SplunkInfoLogger
2017-05-15 17:27:15.246     53 2017-05-15 17:27:15,246 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=Appenders,name=socket
2017-05-15 17:27:15.249     53 2017-05-15 17:27:15,249 Thread-39 DEBUG Registering MBean org.apache.logging.log4j2:type=c1edad85,component=AsyncAppenders,name=SplunkAsync
2017-05-15 17:27:15.253     53 2017-05-15 17:27:15,253 Thread-39 TRACE Using default SystemClock for timestamps.
2017-05-15 17:27:15.255     53 2017-05-15 17:27:15,254 Thread-39 TRACE Using DummyNanoClock for nanosecond timestamps.
2017-05-15 17:27:15.256     53 2017-05-15 17:27:15,256 Thread-39 DEBUG Reconfiguration complete for context[name=c1edad85] at URI C:\ProgramData\IBM\MQSI\common\wsrr\log4j2.xml (org.apache.logging.log4j.core.LoggerContext@c1b7efc0) with optional ClassLoader: null
2017-05-15 17:27:15.257     53 2017-05-15 17:27:15,257 Thread-39 DEBUG Shutdown hook enabled. Registering a new one.
2017-05-15 17:27:15.259     53 2017-05-15 17:27:15,259 Thread-39 DEBUG LoggerContext[name=c1edad85, org.apache.logging.log4j.core.LoggerContext@c1b7efc0] started OK.
2017-05-15 17:27:20.568     49 [15/5/17 17:27:20:558 IST] 00000031 ManagerAdmin  I   TRAS0018I: The trace state has changed. The new trace state is *=info.
2017-05-15 17:29:02.135     56 log4j:WARN No appenders could be found for logger (net.sf.ehcache.config.ConfigurationFactory).
2017-05-15 17:29:02.136     56 log4j:WARN Please initialize the log4j system properly.
2017-05-15 17:29:02.136     56 log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2017-05-15 17:29:07,135 AsyncAppender-SplunkAsync ERROR Unable to write to stream TCP:10.52.214.26:1514 for appender socket: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to TCP:10.52.214.26:1514
2017-05-15 17:29:07.137     60 2017-05-15 17:29:07,136 AsyncAppender-SplunkAsync ERROR An exception occurred processing Appender socket org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to TCP:10.52.214.26:1514
    at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:143)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:197)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.writeByteArrayToManager(AbstractOutputStreamAppender.java:185)
    at org.apache.logging.log4j.core.appender.SocketAppender.directEncodeEvent(SocketAppender.java:424)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:169)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:451)
    at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:404)
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:125)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:171)
    at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:133)
    ... 11 more
2017-05-15 17:29:07.361     60 2017-05-15 17:29:07,360 AsyncAppender-SplunkAsync ERROR Unable to write to stream TCP:10.52.214.26:1514 for appender socket: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to TCP:10.52.214.26:1514
2017-05-15 17:29:07.363     60 2017-05-15 17:29:07,361 AsyncAppender-SplunkAsync ERROR An exception occurred processing Appender socket org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to TCP:10.52.214.26:1514
    at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:143)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.write(OutputStreamManager.java:197)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.writeByteArrayToManager(AbstractOutputStreamAppender.java:185)
    at org.apache.logging.log4j.core.appender.SocketAppender.directEncodeEvent(SocketAppender.java:424)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:169)
    at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:160)
    at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
    at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
    at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(AsyncAppender.java:451)
    at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:404)
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:125)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:171)
    at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:133)
    ... 11 more
2017-05-15 17:29:41.801    247 2017-05-15 17:29:41,801 Log4j2-TcpSocketManager-Reconnector DEBUG Connection to 10.52.214.26:1514 reestablished.
2017-05-15 17:31:37.844    261 2017-05-15 17:31:37,844 Log4j2-TcpSocketManager-Reconnector DEBUG Connection to 10.52.214.26:1514 reestablished.
2017-05-15 17:36:40.838     60 2017-05-15 17:36:40,837 AsyncAppender-SplunkAsync ERROR Unable to write to stream TCP:10.52.214.26:1514 for appender socket: org.apache.logging.log4j.core.appender.AppenderLoggingException: Error writing to TCP:10.52.214.26:1514
2017-05-15 17:37:10.987    293 2017-05-15 17:37:10,986 Log4j2-TcpSocketManager-Reconnector DEBUG Connection to 10.52.214.26:1514 reestablished.

0 个答案:

没有答案
相关问题
最新问题