解析csv文件以删除不需要的列和双引号

时间：2017-02-16 17:09:58

标签： logstash logstash-grok

我有多个csv文件，其中包含如下数据。使用新信息每小时写入这些csv文件。我使用filebeat将这些文件提供给logstash服务器，但是想要配置logstash conf文件来解析csv文件。

"Record Type","Record Code","Broker Name","Broker UUID","EG Name","EG UUID","Message Flow Name","Message Flow UUID","Application Name","Application UUID","Library Name","Library UUID","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Number of Input Messages","Total Size of Input Messages","Average Size of Input Messages","Maximum Size of Input Messages","Minimum Size of Input Messages","Number of Threads in Pool","Time Maximum Number of Threads reached","Total Number of MQ Errors","Total Number of Messages with Errors","Total Number of Errors Processing Messages","Total Number of Time Outs Waiting for Replies to Aggregate Messages","Total Number of Commits","Total Number of Backouts","Accounting Origin" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","00:52:31.599941","2017-02-16 06:52:31.5999","2017-02-16","01:46:17.773842","2017-02-16 07:46:17.7738","0","0","0","0","0","0","0","0","183935","3226073825","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","01:46:17.773949","2017-02-16 07:46:17.7739","2017-02-16","02:46:19.453657","2017-02-16 08:46:19.4536","0","0","0","0","0","0","0","0","193549","3601568195","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","02:46:19.453716","2017-02-16 08:46:19.4537","2017-02-16","03:46:21.184574","2017-02-16 09:46:21.1845","0","0","0","0","0","0","0","0","201382","3601616866","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","03:46:21.184637","2017-02-16 09:46:21.1846","2017-02-16","04:46:22.285130","2017-02-16 10:46:22.2851","0","0","0","0","0","0","0","0","216962","3600985884","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","04:46:22.285240","2017-02-16 10:46:22.2852","2017-02-16","05:46:18.761927","2017-02-16 11:46:18.7619","0","0","0","0","0","0","0","0","211619","3596362373","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","05:46:18.762035","2017-02-16 11:46:18.7620","2017-02-16","06:46:19.935240","2017-02-16 12:46:19.9352","0","0","0","0","0","0","0","0","230385","3601047136","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous" "Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","06:46:19.935316","2017-02-16 12:46:19.9353","2017-02-16","07:46:21.328939","2017-02-16 13:46:21.3289","0","0","0","0","0","0","0","0","231875","3601265537","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous"

不，我想解析这些文件以获取以下内容

timestamp: 2017-02-16 00:52:31 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 timestamp: 2017-02-16 01:46:17 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 timestamp: 2017-02-16 02:46:19 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 timestamp: 2017-02-16 03:46:21 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 timestamp: 2017-02-16 04:46:22 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 timestamp: 2017-02-16 05:46:18 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 timestamp: 2017-02-16 06:46:19 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0

我已使用csv过滤器获取以下内容

{"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"00:52:31.599941","timestamp":"2017-02-16:00:52:31.599941"} {"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"01:46:17.773949","timestamp":"2017-02-16:01:46:17.773949"} {"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"02:46:19.453716","timestamp":"2017-02-16:02:46:19.453716"} {"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"03:46:21.184637","timestamp":"2017-02-16:03:46:21.184637"} {"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"04:46:22.285240","timestamp":"2017-02-16:04:46:22.285240"} {"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"05:46:18.762035","timestamp":"2017-02-16:05:46:18.762035"} {"Message Flow Name":"com.payoffquote.PayOffQuote","Total CPU Time":"0","Record Start Date":"2017-02-16","Total Number of Input Messages":"0","Broker Name":"DEV1","Record Start Time":"06:46:19.935316","timestamp":"2017-02-16:06:46:19.935316"}

但我有问题删除双引号并将它们更改为小写。 你能否告诉我是否可以使用grok获得所需的输出？ timestamp: 2017-02-16 00:52:31 broker: DEV1 eg_name: PayoffQuote message_flow_name: com.payoffquote.PayOffQuote input_messages: 0 cpu_time: 0 ......so on

1 个答案:

答案 0 :(得分：0)

Grok并不真正做替换，只是提取字符串来填充字段。我怀疑你会得到更多的快乐is with the mutate filter。与grok结合使用可以获得您正在寻找的东西。

mutate {
  gsub -> [ 
    "message_flow_name", " ", "_"
  ]
  lowercase => [ "message_flow_name" ]
}

将在字段数据中用下划线替换空格，然后将其小写。

但是，如果您尝试将字段名称设置得更低并且强调，则需要更多工作，而且我不知道如何迭代不使用ruby {}过滤器的字段。这留下了蛮力方法。不漂亮。

mutate {
  rename => {
    "Message Flow Name" => "message_flow_name"
    "Input Message"     => "input_message"
    [...]
    "Accounting Origin" => "accounting_origin"
  }
}
相关问题
最新问题