Question

当我按照flink document通过flink流写入数据时，发生“不可序列化”错误。我使用flink1.6，Elastic-Search-6.4和flink-connector-elasticsearch6。
我的代码就像

@Test
  public void testStringInsert() throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
    env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
    env.enableCheckpointing(100); //
    DataStreamSource<String> input = env.fromCollection(Collections.singleton("testData"));
    List<HttpHost> httpHosts = new ArrayList<>();
    httpHosts.add(new HttpHost("127.0.0.1", 9200, "http"));
    ElasticsearchSink.Builder<String> esSinkBuilder = new ElasticsearchSink.Builder<>(
        httpHosts,
        new ElasticsearchSinkFunction<String> ()  {
          public IndexRequest createIndexRequest(String element) {
            Map<String, String> json = new HashMap<>();
            json.put("data", element);
            return Requests.indexRequest()
                .index("my-index")
                .type("my-type")
                .source(json);
          }
          @Override
          public void process(String element, RuntimeContext ctx, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
          }
        }
    );
    esSinkBuilder.setBulkFlushMaxActions(1);
    input.addSink(esSinkBuilder.build());
    env.execute("test es string insert");
  }

运行上面的代码时，出现异常

java.lang.IllegalArgumentException: The implementation of the provided ElasticsearchSinkFunction is not serializable. The object probably contains or references non-serializable fields.

	at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:139)
	at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.<init>(ElasticsearchSinkBase.java:216)
	at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:71)
	at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink.<init>(ElasticsearchSink.java:60)
	at org.apache.flink.streaming.connectors.elasticsearch6.ElasticsearchSink$Builder.build(ElasticsearchSink.java:208)
	at com.lianlianpay.erebus.erebusaccess.ElasticsearchSinkTest.testStringInsert(ElasticsearchSinkTest.java:151)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

我进行了很多搜索，但仍然感到困惑。我没有将Object传递给ElasticsearchSink，而String除外，并且String毫无疑问是可序列化的。我真的不明白，我的代码或开发环境有什么问题吗？

Answer 1

我刚刚遇到了相同的错误消息。问题在于凭据提供商，应该在RestClientFactory中实例化

Scala中的示例：

class SecuredRestClientFactory (username: String, password: String) extends RestClientFactory {
  override def configureRestClientBuilder(builder: RestClientBuilder): Unit = {
    builder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
      override def customizeHttpClient(httpClientBuilder: HttpAsyncClientBuilder): HttpAsyncClientBuilder = {
        val credentialsProvider = new BasicCredentialsProvider
        credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password))
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)
      }
    })
  }}
esSinkBuilder.setRestClientFactory(new SecuredRestClientFactory(elasticUser,elasticPass))

Answer 2

Flink Elasticsearch Connector 7

请找到我提供的有效且详细的答案 here，它是用Scala编写的。

提供的ElasticsearchSinkFunction的实现不可序列化（flink-connector-elasticsearch6_2.11）

2 个答案: