流利的尾部多行模式捕获日志非常慢。每次捕获日志并重新解析需要8-10分钟

时间:2020-02-27 10:47:17

标签: fluentd

我正在使用流利的beloe管道配置:

<source>
@type tail
@id isc-algo-engine-service
format multiline
multiline_flush_interval 3s
format_firstline /\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}(,|.)\d{3}/
format1 /^(?\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}(,|.)\d{3}) (?.) (?[.]) (?([A-a]lert|ALERT|[D|d]ebug|DEBUG|[N|n]otice|NOTICE|[I|i]nfo|INFO|[W|w]arn?(?:ing)?|WARN?(?:ING)?|ERR?(?:OR)?|[C|c]rit?(?:ical)?|CRIT?(?:ICAL)?|[F|f]atal|FATAL|[S|s]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)) (?.* .* .) (?[\s\S]) (?[\s\S]) : (?.)/
path /srv/isc-algo-engine-service/logs/out-test.log
pos_file /var/log/python2.posg
tag java-online.*
time_format %Y-%m-%dT%H:%M:%S.%NZ
refresh_interval 120
<source>

用例:

如果我使用单行模式,没问题:

2020-02-26T14:16:09+00:00 java-online.srv.isc-algo-engine-service.logs.out-test.log {"timestamp":"2020-02-26 12:23:13,897","hostname":"isc-algo-engine-service-7896997cdc-bvnhb","appname":"[isc-algo-engine-service]","loglevel":"ERROR","class":"utility wrapper 127","process":"[31]","thread":"[140659836589824]","message":"Traceback (most r\necent call last)\njava.lang.NullPointerException: null\n at com.isyscore.os.isclink.data.save.run.emq.EmqHandleRunnable.tagOldUpload(EmqHandleRunnable.java:70)\n at com.isyscore.os.isclink.data.save.run.emq.EmqHandleRunnable.run(EmqHandleRunnable.java:47)\n at com.isyscore.os.isclink.data.save.run.emq.EmqHandleRunnable$$FastClassBySpringCGLIB$$29f375f2.invoke()\n at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)\n at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)\n at com.isyscore.boot.web.monitor.MonitorAspect.doAroundService(MonitorAspect.java:30)\n at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)\n at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)"}

预期的行为: 在3分钟内捕获日志

我的环境: 流利的或td代理版本:流利的1.8.1 Docker映像:fluentd:v1.8.1-debian-2.0

0 个答案:

没有答案