Logstash JDBC适配器：可变到UTF-8？ （mysql弹性导入）

Question

我正在尝试通过Logstash将mysql表导入elasticsearch。一列的类型为“ varbinary”，这会导致以下错误：

[2018-10-10T12:35:54,922][ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error_message=>"\"\\xC3\" from ASCII-8BIT to UTF-8", :error_class=>"LogStash::Json::GeneratorError", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/json.rb:27:in `jruby_dump'", "/usr/share/logstash/vendor/$

我的logstash配置：

input {
  jdbc { 
    jdbc_connection_string => "jdbc:mysql://localhost:3306/xyz"
    # The user we wish to execute our statement as
    jdbc_user => "test"
    jdbc_password => "test"
    # The path to our downloaded jdbc driver
    jdbc_driver_library => "/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    # our query
    statement => "SELECT * FROM x"
    }
  }
output {
  stdout { codec => json_lines }
  elasticsearch {
  "hosts" => "localhost:9200"
  "index" => "x"
  "document_type" => "data"
  }
}

如何将varbinary转换为uft-8？我必须使用特殊的过滤器吗？

Answer 1

好吧...花了几个小时之后，我在发布此问题后立即找到了解决方案：

columns_charset => { "column0" => "UTF8" }

Answer 2

尝试在连接字符串（ characterEncoding = utf8 ）中使用可选内容

jdbc_connection_string => "jdbc:mysql://localhost:3306/xyz?useSSL=false&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&autoReconnect=true"