Question

Is there a way of having the filename of the file being read by logstash as the index name for the output into ElasticSearch?

I am using the following config for logstash.

input{
    file{
        path => "/logstashInput/*"
    }
}
output{
    elasticsearch{
        index => "FromfileX"
    }
}

I would like to be able to put a file e.g. log-from-20.10.2016.log and have it indexed into the index log-from-20.10.2016. Does the logstash input plugin "file" produce any variables for use in the filter or output?

Answer 1

是的，您可以使用path字段和grok字段将文件名解压缩到index字段

  input {
     file {
         path => "/logstashInput/*"
     }
  }
  filter {
     grok {
        match => ["path", "(?<index>log-from-\d{2}\.\d{2}\.\d{4})\.log$" ]
     }
  }
  output{
     elasticsearch {
        index => "%{index}"
     }
  }

Answer 2

input {
    file {
        path => "/home/ubuntu/data/gunicorn.log"
        start_position => "beginning"
    }
}

filter { 
    grok {
        match => {
        "message" => "%{USERNAME:u1} %{USERNAME:u2} \[%{HTTPDATE:http_date}\] \"%{DATA:http_verb} %{URIPATHPARAM:api} %{DATA:http_version}\" %{NUMBER:status_code} %{NUMBER:byte} \"%{DATA:external_api}\" \"%{GREEDYDATA:android_client}\""
        remove_field => ["message"]
       }
    }

    date {
        match => ["http_date", "dd/MMM/yyyy:HH:mm:ss +ssss"]
    } 

    ruby {
        code => "event.set('index_name',event.get('path').split('/')[-1].gsub('.log',''))"
    } 
}
output {
    elasticsearch {
        hosts => ["0.0.0.0:9200"]
        index => "%{index_name}-%{+yyyy-MM-dd}"
        user => "*********************"
        password => "*****************"
    }

    stdout { codec => rubydebug }
}

Logstash input filename as output elasticsearch index

2 个答案: