我遇到了与elasticsearch flink连接器有关的一些问题。 在运行我的程序时,我有很多这样的错误,我很难理解问题出在哪里:
java.lang.NoClassDefFoundError: org/apache/lucene/store/StoreRateLimiting$Type
at org.elasticsearch.index.store.IndexStoreConfig.<clinit>(IndexStoreConfig.java:40)
at org.elasticsearch.common.settings.ClusterSettings.<clinit>(ClusterSettings.java:169)
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:66)
at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:138)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:265)
at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:130)
at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:116)
at org.elasticsearch.transport.client.PreBuiltTransportClient.<init>(PreBuiltTransportClient.java:106)
at org.apache.flink.streaming.connectors.elasticsearch5.Elasticsearch5ApiCallBridge.createClient(Elasticsearch5ApiCallBridge.java:71)
at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.open(ElasticsearchSinkBase.java:272)
at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111)
at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.store.StoreRateLimiting$Type
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 16 more
在pom文件中我添加了:
pom的最后一部分:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.8</version>
<executions>
<execution>
<id>unpack</id>
<!-- executed just before the package phase -->
<phase>prepare-package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<!-- For jarchivelib connector -->
<artifactItem>
<groupId>org.rauschig</groupId>
<artifactId>jarchivelib</artifactId>
<version>${jarchivelib.version}</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/rauschig/**</includes>
</artifactItem>
<!-- For guava library -->
<artifactItem>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>com/google/**</includes>
</artifactItem>
<!-- For cassandra connector -->
<artifactItem>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-cassandra_2.10</artifactId>
<version>${flink.version}</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/flink/**</includes>
</artifactItem>
<artifactItem>
<groupId>com.codahale.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>3.0.2</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>com/codahale/**</includes>
</artifactItem>
<artifactItem>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>${cassandra.version}</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>com/datastax/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch-base_2.10</artifactId>
<version>1.3.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/flink/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-elasticsearch5_2.10</artifactId>
<version>1.3.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/flink/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.client</groupId>
<artifactId>transport</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>6.6.0</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/**</includes>
</artifactItem>
<artifactItem>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.9.9</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/joda/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>transport-netty3-client</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>transport-netty4-client</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>reindex-client</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>percolator-client</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>lang-mustache-client</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>parent-join-client</artifactId>
<version>5.5.1</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/elasticsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>com.carrotsearch</groupId>
<artifactId>hppc</artifactId>
<version>0.7.2</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>com/carrotsearch/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.8.2</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/logging/log4j/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.8.2</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/logging/log4j/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.3</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/**</includes>
</artifactItem>
<artifactItem>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.4.6</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>./target/classes</outputDirectory>
<includes>org/apache/**</includes>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
如果有人可以提供帮助,我将非常高兴。我使用的是flink 1.3.0和elasticsearch5.x!