Question

我需要实现休息服务，该服务将请求中的任何类型数据并通过logstash将其放入elasticsearch。

Spring控制器接收请求正文：

public class CustomData {
    private String component;
    private Object data;
}

数据是来自PUT请求的任何自定义json。我尝试使用net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder

Logstash配置如下：

input {                                                                                                                                                                                                             
        tcp {                                                                                                                                                                                                       
                port => 5000                                                                                                                                                                                        
                codec => json                                                                                                                                                                                               
        }
}


output {
        elasticsearch {
            hosts => "elasticsearch:9200"
            index => "%{indexName}-%{+YYYY.MM.dd}"
        }
}

如您所见，有一个indexName参数 - 我用MDC设置它。但是在消息通过logstash之后，elastic表示我的对象没有映射：

  {
  "appId1-2016.03.15" : {
    "aliases" : { },
    "mappings" : {
      "logs" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "@version" : {
            "type" : "long"
          },
          "host" : {
            "type" : "string"
          },
          "port" : {
            "type" : "long"
          }
        }
      },
      "e1" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "strict_date_optional_time||epoch_millis"
          },
          "@version" : {
            "type" : "long"
          },
          "applicationId" : {
            "type" : "string"
          },
          "host" : {
            "type" : "string"
          },
          "level" : {
            "type" : "string"
          },
          "level_value" : {
            "type" : "long"
          },
          "logger_name" : {
            "type" : "string"
          },
          "message" : {
            "type" : "string"
          },
          "port" : {
            "type" : "long"
          },
          "thread_name" : {
            "type" : "string"
          },
          "type" : {
            "type" : "string"
          },
          "userId" : {
            "type" : "string"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1458053563829",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "w5y7GPd-Sk65gdpBQ_IKow",
        "version" : {
          "created" : "2020099"
        }
      }
    },
    "warmers" : { }
  }
}

只有string类型的消息。

来自logstash的

stdout是：

elasticsearch_1 | [2016-03-16 04:38:49,018][INFO ][cluster.metadata         ] [Ripfire] [json_encoder-events-2016.03.16] create_mapping [ef]
elasticsearch_1 | [2016-03-16 04:38:49,108][INFO ][cluster.metadata         ] [Ripfire] [json_encoder-events-2016.03.16] update_mapping [logs]
elasticsearch_1 | [2016-03-16 04:38:49,174][INFO ][cluster.metadata         ] [Ripfire] [json_encoder-events-2016.03.16] update_mapping [ef]
logstash_1      | 2016-03-16T04:38:48.197Z 10.27.13.228 Initializing Spring FrameworkServlet 'dispatcherServlet'
logstash_1      | 2016-03-16T04:38:48.197Z 10.27.13.228 FrameworkServlet 'dispatcherServlet': initialization started
logstash_1      | 2016-03-16T04:38:48.219Z 10.27.13.228 FrameworkServlet 'dispatcherServlet': initialization completed in 22 ms
logstash_1      | 2016-03-16T04:38:48.412Z 10.27.13.228 {"field1":"value","field2":40000}
logstash_1      | 2016-03-16T04:38:48.423Z 10.27.13.228 {"field1":"value","field2":40000}
logstash_1      | 2016-03-16T04:38:48.457Z 10.27.13.228 
logstash_1      | request payload={
logstash_1      |   "type": "ef",
logstash_1      |   "userId": "ASD",
logstash_1      |   "data": {
logstash_1      |     "field1": "value",
logstash_1      |     "field2": 40000
logstash_1      |   }
logstash_1      | }
logstash_1      | 
logstash_1      | response payload=null

有没有办法根据它的结构来映射data字段？非常感谢您的建议......

elasticsearch中的动态logstash json映射

0 个答案: