示例Apache访问日志

Question

有人可以帮我解决流畅的Apache访问日志解析吗？

我们正在使用Kibana查看日志，但我们无法使用解析器拆分消息部分。下面是我们的示例访问日志，

示例Apache访问日志

127.0.0.1 - - [07/Sep/2017:04:52:19 +0000] "GET /xxxxx/xxxx/xxxx/js/jquery.min.js HTTP/1.1" 200 32775 5113 "domain.test.com" "http://domain.test.com/xxxxx/login.jsp?redirect=http%3A%2F%2Fdomain.test.com%2Fxxxxxx%2Fxxxxxx.jsp" "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"

我们在fluentd.conf中使用以下配置设置将解析器apache访问日志发送到elasticsearch

<source>
  @type tail
  path /var/log/apache2/access.log 
  pos_file /var/log/td-agent/apache-access.log.pos
  tag apache.access
  format apache2
</source>

<filter apache.access>
  @type record_transformer
  <record>
    hostname "#{Socket.gethostname}"
  </record>
</filter>

 <source>
   @type forward
   port 9200
 </source>

 <match *.*>
   @type elasticsearch
   logstash_format true
   host xx.xx.xx.xx 
   port 9200
   index_name fluentd 
   type_name fluentd
 </match>

但我们在kibana中显示的日志不是解析格式。请参阅kibana中显示的示例日志， accesslog_in_kiana_sample

我们期待kibana中的日志类似于以下内容，

_host: 127.0.0.1 - _user: - _time: [07/Sep/2017:04:52:19 +0000] _method: GET _path: /xxxxx/xxxx/xxxx/js/jquery.min.js HTTP/1.1" _code: 200 _size: 32775 _responsetime: 5113 _domain: "domain.test.com" _url: http://domain.test.com/xxxxx/login.jsp?redirect=http%3A%2F%2Fdomain.test.com%2Fxxxxxx%2Fxxxxxx.jsp" _agent: "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.8 Safari/534.34"

我们还尝试在格式

的fluentd.conf文件中添加以下正则表达式

format /^(?<host>[^ ]*(?:,\s+[^ ]+)*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*) (?<responsetime>[^ ]*) "(?<domain>[^\"]*)" "(?<url>[^\"]*)" "(?<agent>[^\"]*)" "(?<jsession>[^\"]*)" "(?<skab>[^\"]*)" "(?<desktopview>[^\"]*)"/

我无法弄清楚上面的错误

在流畅的

示例Apache访问日志

0 个答案: