Question

将Flink配置为可在aws EMR上运行后，我启动了一个scala shell并处理了一些可从kafka读取的代码。当我尝试创建FlinkKafkaConsumer时，出现以下错误：

java.lang.NoClassDefFoundError：org / apache / kafka / common / serialization / ByteArrayDeserializer

我尝试使用：require命令将kafka-clients.jar手动添加到类路径，然后导入org.apache.kafka.common.serialization.ByteArrayDeserializer，仍然出现相同的错误

import java.util.Properties
import java.util.regex.Pattern
import org.apache.flink.api.common.typeinfo.TypeInformation
import org.apache.flink.api.common.serialization.SimpleStringSchema
import org.apache.flink.core.io.SimpleVersionedSerializer
import org.apache.flink.streaming.api.CheckpointingMode
import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.SimpleVersionedStringSerializer
import org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.streaming.api.TimeCharacteristic
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer011
import org.apache.kafka.common.serialization.ByteArrayDeserializer

val properties = new Properties()
properties.setProperty("bootstrap.servers", kafkaEndpoint)
properties.setProperty("zookeeper.connect", zookeeperEndpoint)
properties.setProperty("group.id", "test")
properties.setProperty("auto.offset.reset", "earliest")

val myConsumer = new FlinkKafkaConsumer011[String]("topic1", new SimpleStringSchema(), properties)

val stream = env
    .addSource(myConsumer)
    .print()

我在本地计算机上测试了类似的代码，它可以正常工作，因为我可以看到消息流，但是无法在emr上重现相同的结果。配置emr时是否缺少某些内容？

感谢您的帮助！

如何修复“ java.lang.NoClassDefFoundError：org / apache / kafka / common / serialization / ByteArrayDeserializer”错误？

0 个答案: