我有这个消费者设置正在使用以下代码中来自Kafka主题的消息:
def kafkaSetup(ssc: StreamingContext): InputDStream[MessageAndMetadata[String, String]] = {
val kafkaParams = Map(
"auto.offset.reset" -> "smallest",
"metadata.broker.list" -> config.getString("props.brokerList")
)
val props = new Properties()
props.put("bootstrap.servers", config.getString("props.brokerList"))
props.put("group.id", config.getString("props.groupId"))
props.put("enable.auto.commit", config.getString("props.autoCommit"))
props.put("auto.commit.interval.ms", config.getString("props.autoCommitInterval"));
props.put("session.timeout.ms", config.getString("props.sessionTimeoutInterval"))
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
val consumer = new KafkaConsumer[String, String](props)
consumer.subscribe(config.getString("topic"))
val lines = KafkaUtils.createStream(
ssc,
config.getString("Zookkeeprer"),
config.getString("ConsumerKafkaProps.groupId"),
config.getString("Kafka.topic"),
StorageLevel.MEMORY_AND_DISK_SER_2)
未指定的值参数:kafkaParams:util.Map [String,String],主题:util.Map [String,Integer],storageLevel:StorageLevel