Question

我用以下脚本设置我的河流：

curl -XPUT 'localhost:9200/_river/foo/_meta' -d '{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://...:3306/....",
        "user" : "...",
        "password" : "...",
        "sql" : "SELECT v.id as _id,v.name,v.entrydate, v.link, v.html,v.created AS _created,vc.name AS company, vp.name AS position FROM foo v LEFT JOIN foocompany vc ON vc.id=v.company LEFT JOIN fooposition vp ON vp.id=v.position ",
        "fetchsize" : 100,
        "bulk_size" : 100,
        "max_bulk_requests" : 2,
        "bulk_flush_interval" : "30s",
        "strategy": "simple",
        "poll": "30s",
        "autocommit": true
    }
}'

在这条河流运行一段时间后，我得到一个例外，这可能是因为MySQL服务器本身的配置：

[2014-11-27 16:54:02,301][ERROR][org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:231)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.move(SimpleRiverFlow.java:129)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.run(SimpleRiverFlow.java:88)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:928)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2053)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:406)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:385)
        at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:163)
        at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7472)
        at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:919)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.close(SimpleRiverSource.java:613)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.execute(SimpleRiverSource.java:263)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:227)
        ... 3 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3166)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3620)
        ... 15 more

这里的问题是在我的设置中重新配置MySQL不是一个选项。唉，我必须在其他地方寻求选择。

ElasticSearch是否可以重新建立连接一段时间后再恢复索引？
还有其他吗？如何在不使用JDBC的情况下将ElasticSearch与MySQL联系起来河？

Answer 1

我对Elastic河流有许多头痛。不只是JDBC，还有自定义编写的河流，网络爬虫河等等。

一个重要的注意事项是，河流很快就会被弃用。（Preferred method of indexing bulk data into ElasticSearch?）

我所看到的一个问题是，当Elastic重新启动时，河流并不总能可靠地启动。有时河流根本不会开始，有时它们会开始。非常令人沮丧。

Elastic的官方建议是将流程移到Elastic之外并将数据泵入。

我已经用在Linux上运行的小型C＃应用程序替换了我们所有的JDBC河流，作为同一弹性服务器上的cron作业。效果很好，它更可靠，更容易启动/重启。在Elastic中重建河流对我来说总是很痛苦。

Elasticsearch JDBC河流MySQL连接超时

1 个答案: