我正在使用logstash将日志从DynamoDB推送到ES:
filter {
json {
source => "message"
target => "doc"
}
mutate {
convert => {
"[doc][dynamodb][keys][customerId][N]" => "integer"
"[doc][dynamodb][newImage][callDate][N]" => "integer"
"[doc][dynamodb][newImage][price][S]" => "float"
}
}
date {
match => [ "[doc][dynamodb][newImage][callDate][N]", "UNIX" ]
target => "@timestamp"
}
}
output {
elasticsearch {
hosts => ["localhost"]
codec => "json"
index => "cdr-%{+YYYY.MM.dd}"
document_type => "cdr"
document_id => "%{[doc][dynamodb][keys][uniqueId][S]}"
template_name => "cdr"
template => "/opt/logstash/templates/logstash_dynamodb_template.json"
template_overwrite => true
}
stdout { }
}
实际上,mutate.convert不做任何更改,无论是删除还是添加。
{
"order": 0,
"template": "cdr*",
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"cdr": {
"dynamic": "false",
"_all": {
"enabled": false
},
"properties": {
"doc": {
"properties": {
"dynamodb": {
"properties": {
"keys": {
"properties": {
"customerId": {
"properties": {
"N": {
"type": "long"
}
}
},
"uniqueId": {
"properties": {
"S": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
},
"newImage": {
"properties": {
"callDate": {
"properties": {
"N": {
"type": "date",
"format": "epoch_second"
}
}
},
"direction": {
"properties": {
"S": {
"type": "string",
"index": "not_analyzed"
}
}
},
"disposition": {
"properties": {
"S": {
"type": "string",
"index": "not_analyzed"
}
}
},
"price": {
"properties": {
"S": {
"type": "double"
}
}
},
"uniqueId": {
"properties": {
"S": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
}
}
}
}
}
}
是的,doc.message包含所有描述的字段,但它们未映射。这是ES的截图: 正如您所看到的,只有字符串映射可以正常工作。
查询时出错:No mapping found for [doc.dynamodb.newImage.callDate.N] in order to sort on
有谁知道这种行为的原因是什么?
提示:logstash debug bin/logstash -f filters/logstash-dynamodb.conf --debug
不包含任何错误。
提前感谢任何想法。