使用Spark时如何在Zookeeper中保存偏移ID?

时间:2016-09-23 09:20:04

标签: java apache-spark apache-kafka apache-zookeeper

我正在使用Spark来使用来自Kafka的数据。我正在用Java编写程序。我希望能够在Zookeeper中保存偏移ID,并能够从上次消耗的偏移id重新启动。如何在Zookeeper中保存偏移ID?

SparkConf sparkConf = new SparkConf().setAppName("name");
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, new Duration(2000));
Map<String,String> kafkaParams = new HashMap<>();
kafkaParams.put("zookeeper.connect", "127.0.0.1");
kafkaParams.put("group.id", App.GROUP);
JavaPairReceiverInputDStream<String, EventLog> messages =
  KafkaUtils.createStream(jssc, String.class, EventLog.class, StringDecoder.class, EventLogDecoder.class,
    kafkaParams, topicMap, StorageLevel.MEMORY_AND_DISK_SER_2());

0 个答案:

没有答案