Question

我试图整合kafka和flink。我们的想法是使用kafka队列并使用flink转换数据。我跟随下面提到的例子

https://github.com/abhishek-ch/evolveML/blob/master/flink/kafka-flink-example/pom.xml

这些是我的依赖。

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-core</artifactId>
            <version>0.9.1</version>
           </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.11</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-kafka-0.8_2.11</artifactId>
            <version>1.3.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.11</artifactId>
            <version>1.0.0</version>
        </dependency>

另外，我在项目中包括kafka和flink类如下。

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>3.0.2</version>
            <executions>
                <execution>
                    <id>unpack</id>
                    <!-- executed just before the package phase -->
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>unpack</goal>
                    </goals>
                    <configuration>
                        <artifactItems>
                            <!-- For Flink connector classes -->
                            <artifactItem>
                                <groupId>org.apache.flink</groupId>
                                <artifactId>flink-connector-kafka-0.8_2.11</artifactId>
                                <version>1.3.2</version>
                                <type>jar</type>
                                <overWrite>false</overWrite>
                                <outputDirectory>${project.build.directory}/classes</outputDirectory>
                                <includes>org/apache/flink/**</includes>
                            </artifactItem>
                            <!-- For Kafka API classes -->
                            <artifactItem>
                                <groupId>org.apache.kafka</groupId>
                                <artifactId>kafka_2.11</artifactId>
                                <version>1.0.0</version>
                                <type>jar</type>
                                <overWrite>false</overWrite>
                                <outputDirectory>${project.build.directory}/classes</outputDirectory>
                                <includes>kafka/**</includes>
                            </artifactItem>
                        </artifactItems>
                    </configuration>
                </execution>
            </executions>
       </plugin>

我的消耗kafka队列的Java代码是

  StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

    Map<String, String> map = new HashMap<>();
    map.put("bootstrap.servers", kafka_server);
    map.put("zookeeper.connect", "localhost:40862");
    map.put("group.id", "test");
    map.put("topic", "data");

    // parse user parameters
    ParameterTool parameterTool = ParameterTool.fromMap(map);

    DataStream<String> messageStream = null;
    try {
        messageStream = env.addSource(new org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer082<>(
                parameterTool.getRequired("topic"),
                new SimpleStringSchema(),
                parameterTool.getProperties()));
    } catch (Exception e) {
        LOGGER.error("Error", e);
    }

    // print() will write the contents of the stream to the TaskManager's standard out stream
    // the rebelance call is causing a repartitioning of the data so that all machines
    // see the messages (for example in cases when "num kafka partitions" < "num flink operators"
    messageStream.rebalance().map(new MapFunction<String, String>() {
        private static final long serialVersionUID = -6867736771747690202L;

        @Override
        public String map(String value) throws Exception {
            LOGGER.info("============================" + value);
            return "Kafka and Flink says: " + value;
        }
    }).print();

    try {
        env.execute();
    } catch (Exception e) {
        e.printStackTrace();
    }

此代码示例来自我之前提到的github项目。此代码在tomcat中部署的war文件中运行。

运行此代码时出现以下错误。

      Unrecoverable error java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer082

我在战争摘录中提到了类。我试图弄清楚如何解决这个问题。任何帮助或建议高度赞赏。

Answer 1

NoClassDefFoundError经常暗示版本/依赖性问题，实际上你的依赖关系有些混乱。

您正在从1.3.2（当前版本）和0.9.1（一个相当古老的版本）导入Flink依赖项。 Flink Kafka连接器适用于Kafka 0.8，但你可以使用Kafka 1.0.0依赖。

Answer 2

您应该修改flink-streaming-core依赖项，而不是依赖于flink-streaming-java_2.11，版本1.3.2。（几年前，flink-streaming-core被重命名为flink-streaming-java和flink-streaming-scala。）

此外，flink-connector-kafka-0.8_2.11适用于Kafka版本0.8.x，而您将它与Kafka版本1.0.0相结合。我建议你删除kafka_2.11依赖，并依赖maven传递包含正确版本的kafka jar。

尝试将Kafka与Flink集成时出现NoClassDefFoundError

2 个答案: