Elasticsearch将多个json数据插入嵌套字段

时间:2019-11-13 03:26:48

标签: elasticsearch logstash

我在将新数组数据添加到嵌套字段时遇到问题。我的输入是Rabbitmq queue(json)。

我的索引映射如下

{
  "txn" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "body" : {
          "properties" : {            
            "transactionAmount" : {
              "type" : "float"
            },
            "entryTimestamp" : {
              "type" : "date"
            },
            "reason" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "header" : {
          "properties" : {
            "counter" : {
              "type" : "long"
            },
            "serialNum" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "response" : {
          "type" : "nested"
        }
      }
    }
  }
}

我消耗的数据和索引为“ txn”的数据如下。我根据serialNum设置了docID。

{
    "header": {
        "serialNum": "ABC",
        "counter": 2
    },
    "body": {
        "transactionAmount": 30.50,
        "reason": "Transaction Record success",
        "entryTimestamp": "2019-11-13T10:56:28.160+08:00"
    }
}

以及如下所示的logstash

# mgr
input {
  rabbitmq {
         host => "localhost"
         vhost => "/v.uat"
         queue => "txn"
         heartbeat => 30
         durable => true
         password => "abcde"
         user => "abcde"
    }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    document_id => "%{[header][serialNum]}"
    index => "txn"
    }
}

问题是,我需要从另一个队列中消费并更新“响应”嵌套字段。将整个json更新为“响应”字段。因此,如果输入具有2个json数据,则它是“响应”嵌套字段中的2个json数据数组。 如果输入有2个数据,我该如何更新'txn'索引?

要在“响应”字段中更新的示例数据

Sample 1
{
    "header": {
        "serialNum": "ABC",
        "counter": 2,
        "version": "1.0"
    },
    "body": {
        "responseCode": "0",
        "responseDescription": "Transaction Record success",
        "commission": 0.0657,
        "planDescription": "STANDARD",
        "postedDate": "2019-10-13T10:56:28.160+08:00"
    }
}

Sample 2
{
    "header": {
        "serialNum": "ABC",
        "counter": 3,
        "version": "1.0"
    },
    "body": {
        "responseCode": "2",
        "responseDescription": "Transaction Record failed",
        "commission": 0.0657,
        "planDescription": "STANDARD",
        "postedDate": "2019-10-12T09:12:22.190+08:00"
    }
}

希望您可以共享logstash脚本来解决我的问题。 真的需要您的帮助

0 个答案:

没有答案