Question

我正在尝试在AWS实例上设置kafka-connect-cassandra。

我在plugin.path文件中设置了connect-avro-distributed.properties：

plugin.path=/home/ubuntu/kafka_2.11-1.0.0/plugins

我有kafka-connect-cassandra-1.0.0-1.0.0-all.jar：

/home/ubuntu/kafka_2.11-1.0.0/plugins/lib

这是追溯：

[2018-02-18 10:28:33,268] INFO Kafka Connect distributed worker initializing ... (org.apache.kafka.connect.cli.ConnectDistributed:60)
[2018-02-18 10:28:33,278] INFO WorkerInfo values: 
    jvm.args = -Xmx256M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/log/kafka, -Dlog4j.configuration=file:/etc/kafka/connect-log4j.properties
    jvm.spec = Oracle Corporation, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_151, 25.151-b12
    jvm.classpath = /home/ubuntu/kafka_2.11-1.0.0/plugins:/usr/share/java/kafka/jackson-jaxrs-json-provider-2.9.1.jar:/usr/share/java/kafka/paranamer-2.7.jar:/usr/share/java/kafka/rocksdbjni-5.7.3.jar:/usr/share/java/kafka/jackson-jaxrs-base-2.9.1.jar:/usr/share/java/kafka/jetty-servlets-9.2.22.v20170606.jar:/usr/share/java/kafka/kafka-clients-1.0.0-cp1.jar:/usr/share/java/kafka/xz-1.5.jar:/usr/share/java/kafka/commons-lang3-3.1.jar:/usr/share/java/kafka/jetty-security-9.2.22.v20170606.jar:/usr/share/java/kafka/httpclient-4.5.2.jar:/usr/share/java/kafka/jackson-core-asl-1.9.13.jar:/usr/share/java/kafka/connect-json-1.0.0-cp1.jar:/usr/share/java/kafka/connect-runtime-1.0.0-cp1.jar:/usr/share/java/kafka/jersey-container-servlet-2.25.1.jar:/usr/share/java/kafka/commons-collections-3.2.1.jar:/usr/share/java/kafka/connect-transforms-1.0.0-cp1.jar:/usr/share/java/kafka/jersey-common-2.25.1.jar:/usr/share/java/kafka/zookeeper-3.4.10.jar:/usr/share/java/kafka/scala-library-2.11.11.jar:/usr/share/java/kafka/jackson-core-2.9.1.jar:/usr/share/java/kafka/argparse4j-0.7.0.jar:/usr/share/java/kafka/maven-artifact-3.5.0.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-test-sources.jar:/usr/share/java/kafka/commons-codec-1.9.jar:/usr/share/java/kafka/kafka-connect-cassandra-1.0.0-1.0.0-all.jar:/usr/share/java/kafka/httpcore-4.4.4.jar:/usr/share/java/kafka/hk2-utils-2.5.0-b32.jar:/usr/share/java/kafka/connect-api-1.0.0-cp1.jar:/usr/share/java/kafka/javassist-3.21.0-GA.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-sources.jar:/usr/share/java/kafka/support-metrics-client-4.0.0.jar:/usr/share/java/kafka/commons-compress-1.8.1.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-scaladoc.jar:/usr/share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/share/java/kafka/jersey-media-jaxb-2.25.1.jar:/usr/share/java/kafka/kafka-streams-1.0.0-cp1.jar:/usr/share/java/kafka/zkclient-0.10.jar:/usr/share/java/kafka/hk2-locator-2.5.0-b32.jar:/usr/share/java/kafka/slf4j-api-1.7.25.jar:/usr/share/java/kafka/support-metrics-common-4.0.0.jar:/usr/share/java/kafka/kafka.jar:/usr/share/java/kafka/jersey-server-2.25.1.jar:/usr/share/java/kafka/jackson-module-jaxb-annotations-2.9.1.jar:/usr/share/java/kafka/jetty-io-9.2.22.v20170606.jar:/usr/share/java/kafka/kafka-log4j-appender-1.0.0-cp1.jar:/usr/share/java/kafka/avro-1.8.2.jar:/usr/share/java/kafka/jackson-annotations-2.9.1.jar:/usr/share/java/kafka/guava-20.0.jar:/usr/share/java/kafka/hk2-api-2.5.0-b32.jar:/usr/share/java/kafka/lz4-java-1.4.jar:/usr/share/java/kafka/reflections-0.9.11.jar:/usr/share/java/kafka/commons-digester-1.8.1.jar:/usr/share/java/kafka/slf4j-log4j12-1.7.25.jar:/usr/share/java/kafka/jersey-client-2.25.1.jar:/usr/share/java/kafka/commons-lang3-3.5.jar:/usr/share/java/kafka/jackson-mapper-asl-1.9.13.jar:/usr/share/java/kafka/javax.annotation-api-1.2.jar:/usr/share/java/kafka/snappy-java-1.1.4.jar:/usr/share/java/kafka/javax.inject-2.5.0-b32.jar:/usr/share/java/kafka/jackson-databind-2.9.1.jar:/usr/share/java/kafka/jetty-http-9.2.22.v20170606.jar:/usr/share/java/kafka/kafka-streams-examples-1.0.0-cp1.jar:/usr/share/java/kafka/plexus-utils-3.0.24.jar:/usr/share/java/kafka/metrics-core-2.2.0.jar:/usr/share/java/kafka/connect-file-1.0.0-cp1.jar:/usr/share/java/kafka/kafka-tools-1.0.0-cp1.jar:/usr/share/java/kafka/jersey-guava-2.25.1.jar:/usr/share/java/kafka/commons-logging-1.2.jar:/usr/share/java/kafka/jetty-server-9.2.22.v20170606.jar:/usr/share/java/kafka/validation-api-1.1.0.Final.jar:/usr/share/java/kafka/jetty-continuation-9.2.22.v20170606.jar:/usr/share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/share/java/kafka/httpmime-4.5.2.jar:/usr/share/java/kafka/log4j-1.2.17.jar:/usr/share/java/kafka/jopt-simple-5.0.4.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-javadoc.jar:/usr/share/java/kafka/javax.ws.rs-api-2.0.1.jar:/usr/share/java/kafka/jersey-container-servlet-core-2.25.1.jar:/usr/share/java/kafka/jetty-servlet-9.2.22.v20170606.jar:/usr/share/java/kafka/jetty-util-9.2.22.v20170606.jar:/usr/share/java/kafka/commons-validator-1.4.1.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1.jar:/usr/share/java/kafka/javax.inject-1.jar:/usr/share/java/kafka/commons-beanutils-1.8.3.jar:/usr/share/java/kafka/javassist-3.20.0-GA.jar:/usr/share/java/kafka/aopalliance-repackaged-2.5.0-b32.jar:/usr/share/java/kafka/kafka_2.11-1.0.0-cp1-test.jar:/usr/share/java/confluent-common/zookeeper-3.4.10.jar:/usr/share/java/confluent-common/common-metrics-4.0.0.jar:/usr/share/java/confluent-common/build-tools-4.0.0.jar:/usr/share/java/confluent-common/zkclient-0.10.jar:/usr/share/java/confluent-common/slf4j-api-1.7.25.jar:/usr/share/java/confluent-common/common-utils-4.0.0.jar:/usr/share/java/confluent-common/netty-3.10.5.Final.jar:/usr/share/java/confluent-common/common-config-4.0.0.jar:/usr/share/java/confluent-common/jline-0.9.94.jar:/usr/share/java/confluent-common/log4j-1.2.17.jar:/usr/share/java/kafka-serde-tools/kafka-json-serializer-4.0.0.jar:/usr/share/java/kafka-serde-tools/paranamer-2.7.jar:/usr/share/java/kafka-serde-tools/xz-1.5.jar:/usr/share/java/kafka-serde-tools/jackson-core-asl-1.9.13.jar:/usr/share/java/kafka-serde-tools/jackson-core-2.9.1.jar:/usr/share/java/kafka-serde-tools/kafka-connect-avro-converter-4.0.0.jar:/usr/share/java/kafka-serde-tools/commons-compress-1.8.1.jar:/usr/share/java/kafka-serde-tools/avro-1.8.2.jar:/usr/share/java/kafka-serde-tools/jackson-annotations-2.9.1.jar:/usr/share/java/kafka-serde-tools/kafka-schema-registry-client-4.0.0.jar:/usr/share/java/kafka-serde-tools/kafka-avro-serializer-4.0.0.jar:/usr/share/java/kafka-serde-tools/jackson-mapper-asl-1.9.13.jar:/usr/share/java/kafka-serde-tools/jackson-databind-2.9.1.jar:/usr/share/java/kafka-serde-tools/snappy-java-1.1.1.3.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-json-provider-2.9.1.jar:/usr/bin/../share/java/kafka/paranamer-2.7.jar:/usr/bin/../share/java/kafka/rocksdbjni-5.7.3.jar:/usr/bin/../share/java/kafka/jackson-jaxrs-base-2.9.1.jar:/usr/bin/../share/java/kafka/jetty-servlets-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/kafka-clients-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/xz-1.5.jar:/usr/bin/../share/java/kafka/commons-lang3-3.1.jar:/usr/bin/../share/java/kafka/jetty-security-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/httpclient-4.5.2.jar:/usr/bin/../share/java/kafka/jackson-core-asl-1.9.13.jar:/usr/bin/../share/java/kafka/connect-json-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/connect-runtime-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-2.25.1.jar:/usr/bin/../share/java/kafka/commons-collections-3.2.1.jar:/usr/bin/../share/java/kafka/connect-transforms-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/jersey-common-2.25.1.jar:/usr/bin/../share/java/kafka/zookeeper-3.4.10.jar:/usr/bin/../share/java/kafka/scala-library-2.11.11.jar:/usr/bin/../share/java/kafka/jackson-core-2.9.1.jar:/usr/bin/../share/java/kafka/argparse4j-0.7.0.jar:/usr/bin/../share/java/kafka/maven-artifact-3.5.0.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-test-sources.jar:/usr/bin/../share/java/kafka/commons-codec-1.9.jar:/usr/bin/../share/java/kafka/kafka-connect-cassandra-1.0.0-1.0.0-all.jar:/usr/bin/../share/java/kafka/httpcore-4.4.4.jar:/usr/bin/../share/java/kafka/hk2-utils-2.5.0-b32.jar:/usr/bin/../share/java/kafka/connect-api-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/javassist-3.21.0-GA.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-sources.jar:/usr/bin/../share/java/kafka/support-metrics-client-4.0.0.jar:/usr/bin/../share/java/kafka/commons-compress-1.8.1.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-scaladoc.jar:/usr/bin/../share/java/kafka/javax.servlet-api-3.1.0.jar:/usr/bin/../share/java/kafka/jersey-media-jaxb-2.25.1.jar:/usr/bin/../share/java/kafka/kafka-streams-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/zkclient-0.10.jar:/usr/bin/../share/java/kafka/hk2-locator-2.5.0-b32.jar:/usr/bin/../share/java/kafka/slf4j-api-1.7.25.jar:/usr/bin/../share/java/kafka/support-metrics-common-4.0.0.jar:/usr/bin/../share/java/kafka/kafka.jar:/usr/bin/../share/java/kafka/jersey-server-2.25.1.jar:/usr/bin/../share/java/kafka/jackson-module-jaxb-annotations-2.9.1.jar:/usr/bin/../share/java/kafka/jetty-io-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/kafka-log4j-appender-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/avro-1.8.2.jar:/usr/bin/../share/java/kafka/jackson-annotations-2.9.1.jar:/usr/bin/../share/java/kafka/guava-20.0.jar:/usr/bin/../share/java/kafka/hk2-api-2.5.0-b32.jar:/usr/bin/../share/java/kafka/lz4-java-1.4.jar:/usr/bin/../share/java/kafka/reflections-0.9.11.jar:/usr/bin/../share/java/kafka/commons-digester-1.8.1.jar:/usr/bin/../share/java/kafka/slf4j-log4j12-1.7.25.jar:/usr/bin/../share/java/kafka/jersey-client-2.25.1.jar:/usr/bin/../share/java/kafka/commons-lang3-3.5.jar:/usr/bin/../share/java/kafka/jackson-mapper-asl-1.9.13.jar:/usr/bin/../share/java/kafka/javax.annotation-api-1.2.jar:/usr/bin/../share/java/kafka/snappy-java-1.1.4.jar:/usr/bin/../share/java/kafka/javax.inject-2.5.0-b32.jar:/usr/bin/../share/java/kafka/jackson-databind-2.9.1.jar:/usr/bin/../share/java/kafka/jetty-http-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/kafka-streams-examples-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/plexus-utils-3.0.24.jar:/usr/bin/../share/java/kafka/metrics-core-2.2.0.jar:/usr/bin/../share/java/kafka/connect-file-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/kafka-tools-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/jersey-guava-2.25.1.jar:/usr/bin/../share/java/kafka/commons-logging-1.2.jar:/usr/bin/../share/java/kafka/jetty-server-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/validation-api-1.1.0.Final.jar:/usr/bin/../share/java/kafka/jetty-continuation-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/osgi-resource-locator-1.0.1.jar:/usr/bin/../share/java/kafka/httpmime-4.5.2.jar:/usr/bin/../share/java/kafka/log4j-1.2.17.jar:/usr/bin/../share/java/kafka/jopt-simple-5.0.4.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-javadoc.jar:/usr/bin/../share/java/kafka/javax.ws.rs-api-2.0.1.jar:/usr/bin/../share/java/kafka/jersey-container-servlet-core-2.25.1.jar:/usr/bin/../share/java/kafka/jetty-servlet-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/jetty-util-9.2.22.v20170606.jar:/usr/bin/../share/java/kafka/commons-validator-1.4.1.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1.jar:/usr/bin/../share/java/kafka/javax.inject-1.jar:/usr/bin/../share/java/kafka/commons-beanutils-1.8.3.jar:/usr/bin/../share/java/kafka/javassist-3.20.0-GA.jar:/usr/bin/../share/java/kafka/aopalliance-repackaged-2.5.0-b32.jar:/usr/bin/../share/java/kafka/kafka_2.11-1.0.0-cp1-test.jar:/usr/bin/../share/java/confluent-support-metrics/*:/usr/share/java/confluent-support-metrics/*
    os.spec = Linux, amd64, 4.4.0-1049-aws
    os.vcpus = 2
 (org.apache.kafka.connect.runtime.WorkerInfo:71)
[2018-02-18 10:28:33,279] INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed:69)
[2018-02-18 10:28:33,290] INFO Loading plugin from: /home/ubuntu/kafka_2.11-1.0.0/plugins/lib (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:179)
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
    at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
    at org.reflections.Reflections.<init>(Reflections.java:126)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:258)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)

jvm.classpath中的第一个条目是kafka-connect-cassandra.jar内plugin/lib所在的位置。

guava.jars：

这些是番石榴罐子的路径。我应该在哪里放置kafka-connect-cassandra.jar，还是应该删除这些罐子？

/usr/share/java/kafka-connect-elasticsearch/guava-18.0.jar
/usr/share/java/kafka-connect-storage-common/guava-14.0.1.jar
/usr/share/java/kafka/guava-20.0.jar
/home/ubuntu/cassandra/apache-cassandra-3.11.1/lib/guava-18.0.jar

请帮助我。

Answer 1

Connect框架目前带有guava-20.0.jar以便使用Reflections并能够扫描plugin.path的Connect插件。

如果您使用其中一个bin脚本connect-distributed或connect-standalone启动Connect，则

Connect框架的依赖项将添加到您的CLASSPATH中。

然而，某种程度上，旧版本的guava首先存在于您的CLASSPATH中（可能是因为您已将CLASSPATH设置为包含此类依赖项），然后Connect无法使用番石榴版本扫描您的plugin.path < 20。这就是你的例外所表明的。 Connect无法使用它在CLASSPATH中找到的guava版本扫描plugin.path。

您的plugin.path应该包含您的插件，每个插件都在其自己的目录中。例如，与上面的内容类似：

plugin.path=/home/ubuntu/connect

/home/ubuntu/connect/kafka_2.11-1.0.0/和

/home/ubuntu/connect/kafka-connect-cassandra-1.0.0-1.0.0-all.jar

根据您的示例，上面的布局用于表明插件不应包含其他插件，但不建议将plugin.path设置为用户的主目录。

Answer 2

这是一个类路径问题。看起来你可能在类路径中有一个不兼容的番石榴版本？如果您的插件路径不包含此方法，则说明这是一个连接器打包问题。如果是，那么你可能有两个版本。使用find命令仔细检查插件路径，以便在消息中检查该类的所有jar作为第一步。最终，您需要弄清楚连接器所期望的依赖版本，并将该版本和该版本添加到插件路径中。

Kafka Connect java.lang.NoSuchMethodError：com.google.common.collect.Sets $ SetView.iterator（）Lcom / google / common / collect / UnmodifiableIterator;

2 个答案: