Question

我的英语不好，对不起......

这些天，我总是在AWS EC2上测试td-agent（Amazon Linux AMI版本2014.03）。从nginx和输出日志到S3的输入日志都没问题，这非常好！

然后我希望praser登录到json，当我制作源格式为“nginx”时，它说

"  2015-03-31 00:12:24 +0800 [warn]: pattern not match: "95.218.58.120 - [2015-03-30T16:11:40+08:00] \"GET /provider.txt HTTP/1.1\" 200 726 \"-\" \"FreePP 2.9.5.159 (iPhone; iPhone OS 8.1.3; ar_SA) [65075070]\" "

我认为mybe我的nginx日志不是默认的。然后，我为此做了regexp，

format /^(?<remote>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: (?<path>[^\"]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"?$/

我的日志是： 178.152.203.39 - [2015-03-31T00:18:10+08:00] "GET /provider.txt HTTP/1.1" 200 726 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9300 Build/JSS15J）“

重新启动td-agent，td-agent日志似乎OK。但是没有找到输出，而且我的cpu 100％，任何人都可以帮帮我吗？非常感谢!!!

top - 00:26:17 up 4 days,  1:10,  2 users,  load average: 0.20, 0.35, 0.57
Tasks:  82 total,   3 running,  79 sleeping,   0 stopped,   0 zombie
Cpu(s): 48.3%us,  2.2%sy,  0.0%ni, 47.5%id,  0.0%wa,  0.3%hi,  1.7%si,  0.0%st
Mem:   7401596k total,  6612324k used,   789272k free,   151200k buffers
Swap:        0k total,        0k used,        0k free,  5911168k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                    
 4724 td-agent  20   0  222m  23m 2800 R 96.5  0.3   0:11.70 ruby                                                                                       
  473 nginx     20   0  119m  12m 1668 S  3.7  0.2   5:58.61 nginx



 2015-03-31 00:26:04 +0800 [info]: process finished code=0
2015-03-31 00:26:05 +0800 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-03-31 00:26:05 +0800 [info]: starting fluentd-0.10.61
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-mongo' version '0.7.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-s3' version '0.5.6'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-td' version '0.10.26'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-03-31 00:26:05 +0800 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-03-31 00:26:05 +0800 [info]: gem 'fluentd' version '0.10.61'
2015-03-31 00:26:05 +0800 [info]: using configuration file: <ROOT>
  <source>
    type tail
    format /^(?<remote>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: (?<path>[^\"]*)+\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"?$/
    time_format %d/%b/%YT%H:%M:%S%z
    path /var/log/nginx/provider.access.log
    log_level debug
    pos_file /var/log/td-agent/access.log.pos
    tag s3.nginx.access
  </source>
  <match s3.nginx.access>
    type file
    path /var/log/td-agent/nginxAccess
    time_slice_format %Y%m%d
    time_slice_wait 10m
    format single_value
  </match>
</ROOT>
2015-03-31 00:26:05 +0800 [info]: adding source type="tail"
2015-03-31 00:26:05 +0800 [info]: adding match pattern="s3.nginx.access" type="file"
2015-03-31 00:26:05 +0800 [info]: following tail of /var/log/nginx/provider.access.log

Answer 1

重新启动td-agent，td-agent日志似乎没问题。但是没有找到输出，

从您的配置中，您在文件输出中使用single_value。 single_value几乎适用于none解析器。如果您想使用single_value，则应在尾部输入中使用none。

请参阅本文中的格式部分：http://docs.fluentd.org/articles/out_file#format

Answer 2

正则表达式有些错误，我修改它

格式/ ^（？[^] ）（？[^] ）[（？[^]] ）]“（？\ S +）（？[^”] < / em>）“（？[^] ）（？[^] ）”（？[^ \“] ）”“（？[^ \”] ） “/

pb是解散的。谢谢！

td-agent praser nginx log cpu 100％

2 个答案: