Question

您好我在lostash 2.X版本中使用以下scirp文件我在MySQL数据库表中有超过186000条记录，但在运行此.conf文件时，只有一个文档正在加载弹性搜索索引。

input {
        jdbc {
            jdbc_connection_string => "jdbc:mysql://localhost/elasticsearch"
            jdbc_user => "root"
            jdbc_password => "empower"
            #jdbc_validate_connection => true
            jdbc_driver_library => "/home/wtc082/Documents/com.mysql.jdbc_5.1.5.jar"
            jdbc_driver_class => "com.mysql.jdbc.Driver"
            statement => "SELECT * FROM index_part_content_local;"
            #schedule => "* * * * *"
            #codec => "json"
        }
    }

output {
    elasticsearch {
        index => "mysqltest"
        document_type => "mysqltest_type"
        document_id => "%{id}"
        hosts => "localhost:9200"
    }
}

当我使用此查询时，只有一个文档是索引

GET mysqltest/_search
{
  "query": {
    "match_all": {}
  }
}




{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "mysqltest",
        "_type": "mysqltest_type",
        "_id": "%{id}",
        "_score": 1,
        "_source": {
          "partnum": "",
          "property1": "",
          "property2": "",
          "color": "",
          "size": "",
          "dim": "",
          "thumburl": "",
          "catid": "6575",
          "subcatid": "3813",
          "termid": "31999",
          "longdesc": "<ul><li>Equipment and Parts<li>GC32-XD Parts<li>D/V Lock Plate Screw</ul>",
          "hier1desc": "Heavy Duty Tools / Equipment",
          "hier2desc": "Other Heavy Duty Equipment",
          "hier3desc": "Hose Crimping Equipment & Accessories",
          "aaiabrandid": "BBSC",
          "aaiabrandname": "Gates",
          "brandimageurl": "es-logo-sm.jpg",
          "linecode": "GAT",
          "descrp": "D/V Lock Plate Screw",
          "@version": "1",
          "@timestamp": "2016-12-20T09:16:40.075Z"
        }
      }
    ]
  }
}

Answer 1

好的，您可以看到文档的ID是逐字值"%{id}"，这意味着显然您的数据库中没有任何id列以及您的所有记录数据库在同一文档ID下编制索引，因此您只能看到一个文档。

在elasticsearch输出中，您需要确保使用作为表格主键的字段

    document_id => "%{PRIMARY_KEY}"

修复它，这将有效。

从MySQL To Elasticsearch加载数据

1 个答案: