Question

我有这个logstash配置文件，查询返回10547但在elasticsearch中只看到10500。更大的数据显然会产生更多的损失。

为什么？

input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string => "jdbc:mysql://outsystemsdemomysql.cxxhlwetgta3.us-east-1.rds.amazonaws.com:3306/outsystems1"
        # The user we wish to execute our statement as
        jdbc_user => "xxxxx"
        jdbc_password => "xxxxx"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "mysql-connector-java-5.1.40-bin.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        # our query
        statement => "SELECT OSUSR_QP0_CONTACT.ID,
 CASE WHEN OSUSR_QP0_CONTACT.NAME='' THEN 'Unknown' ELSE OSUSR_QP0_CONTACT.NAME END name,
OSUSR_QP0_CONTACT.TITLE,OSUSR_QP0_CONTACT.PHONE,OSUSR_QP0_CONTACT.ADDRESS,OSUSR_QP0_CONTACT.EMAIL,OSUSR_QP0_ORGANIZATIONTYPE.AKA OrgType
FROM OSUSR_QP0_CONTACT
LEFT JOIN OSUSR_qp0_Organization ON (OSUSR_QP0_CONTACT.ORGANIZATIONID=OSUSR_qp0_Organization.ID)
LEFT JOIN OSUSR_QP0_ORGANIZATIONTYPE ON (OSUSR_qp0_Organization.TYPE=OSUSR_QP0_ORGANIZATIONTYPE.ID)
WHERE  OSUSR_QP0_CONTACT.ARCHIVED = 0"
    }
}
output {
    amazon_es {
        hosts => ["search-prochamps-nzv5olnw2mloxwiqd7s3o557we.us-east-1.es.amazonaws.com"]
        region => "us-east-1"
        # aws_access_key_id, aws_secret_access_key optional if instance profile is configured
        aws_access_key_id => "xxxxx"
        aws_secret_access_key => "xxxx"
        index => "crm"
        document_type => "contacts"
        document_id => "%{id}"

    }


}

Answer 1

尝试一些调试：

在logstash中使用stdout {}输出，并查看它处理的文档数量。
检查logstash和elasticsearch日志以查看是否报告了任何错误（例如，映射中的类型不匹配会导致记录丢失）。

logstash和elasticsearch缺少数据

1 个答案: