Question

基于Storm文档支持的KafkaSpout实现基于旧的消费者API。我注意到外部包有另一个名为storm-kafka-client的实现。

https://github.com/apache/storm/tree/master/external/storm-kafka-client

目前还不清楚1.0.1中的新客户端版本是否已准备就绪。有没有人有运行它的经验？

Answer 1

我在Storm邮件列表中发布了同样的问题。新的API已准备就绪。我们应该使用1.x分支。我计划用

进行测试

<!-- https://mvnrepository.com/artifact/org.apache.storm/storm-kafka-client -->

<dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-kafka-client</artifactId>
    <version>1.0.1</version>
</dependency>

将更新进度。

Answer 2

以下代码对我来说很有用!!!

public TopologyBuilder myTopology() {

    TopologyBuilder builder = new TopologyBuilder();

    try {       
        KafkaSpoutConfig<String, String> kafkaSpoutConfig = getKafkaSpoutConfig("KAFKA_IP:9092", KAFKA_TOPIC);          
        KafkaSpout kafkaSpout = new KafkaSpout<>(kafkaSpoutConfig);
        builder.setSpout("kafkaSpout", kafkaSpout, 2 * 2);
        builder.setBolt("Bolt-1", new TestBolt(), parallelism).shuffleGrouping("kafkaSpout", KAFKA_TOPIC);
    } catch (Exception ex) {

    }

    return builder;
}

配置Spout。

protected KafkaSpoutConfig<String, String> getKafkaSpoutConfig(String bootstrapServers ,String topic) {
    ByTopicRecordTranslator<String, String> trans = new ByTopicRecordTranslator<>(
        (r) -> new Values(r.topic(), r.partition(), r.offset(), r.key(), r.value()),
        new Fields("topic", "partition", "offset", "key", "value"), topic);

    Builder<String, String> builder = KafkaSpoutConfig.builder(bootstrapServers, new String[]{topic});

    return builder.setProp(ConsumerConfig.GROUP_ID_CONFIG, topic)
        .setProcessingGuarantee(ProcessingGuarantee.AT_LEAST_ONCE)
        .setRetry(getRetryService())
        .setRecordTranslator(trans)
        .setOffsetCommitPeriodMs(10_000)
        .setFirstPollOffsetStrategy(UNCOMMITTED_EARLIEST)
        .setMaxUncommittedOffsets(1000)
        .build();
}

配置失败消息retyr logic

protected KafkaSpoutRetryService getRetryService() {
    return new KafkaSpoutRetryExponentialBackoff(TimeInterval.microSeconds(500),
        TimeInterval.milliSeconds(2), Integer.MAX_VALUE, TimeInterval.seconds(10));
 }

Answer 3

您可以对风暴1.1.0使用以下maven依赖关系

  <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
         <version>1.1.0</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

 <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.10.0.0</version>
    </dependency>
    <dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-kafka</artifactId>
    <version>1.0.0</version>
</dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
         <version>0.9.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

您可能会面临更多依赖性问题，您可以通过添加所需的jar来解决这些问题。

此外，java代码中的依赖项将从org.backtype.storm.XXXXX更改为org.apache.storm.XXXXX

来自Storm 1.0.1的Storm-Kafka-client

3 个答案: