Question

我正在运行一个Storm Topology，它正在从AWS上获取Kafka的推文 Ubuntu Server 14.04具有4个节点的LTS实例 - Nimbus，Supervisor，Kafka-Zookeeper节点，Zookeeper（用于Storm集群）。我的Storm UI已启动并运行，我可以提交拓扑。我有两个经纪人，但我只使用broker.id = 0。我在主题下面有推文。我的kafka服务器运行正常。

我用这种方式创建了kafka主题：
bin / kafka-topics.sh --create --zookeeper localhost：2181 / kafka --rerelication-factor 1 --partitions 1 --topic twitter1

我感到困惑的是：

SpoutConfig kafkaConfig = new SpoutConfig(kafkaHosts, topicName+"-0", "/kafka", topicName+"-0");

我认为我的错误正在从这一点萌芽。完整的代码是：

import org.apache.storm.tuple.Fields;
import org.apache.storm.kafka.BrokerHosts;
import org.apache.storm.kafka.KafkaSpout;
import org.apache.storm.kafka.SpoutConfig;
import org.apache.storm.kafka.ZkHosts;
import java.util.Arrays;
import org.apache.storm.Config;
import org.apache.storm.StormSubmitter;
import org.apache.storm.spout.SchemeAsMultiScheme;
import org.apache.storm.topology.TopologyBuilder;
import org.apache.storm.kafka.StringScheme;

public class TwitterTopology{
    public static void main(String[] args) {
        String topicName = "twitter1";
        String topologyName = args[0];
        String kafkaIp = "xxx.31.xxx.207"; //hiding the IPs here. This is the IP for my kafka-zk node. Is this ok?
        String nimbusHost = "xxx.31.xxx.70";
        String kafkaHost = kafkaIp + ":9092";
        BrokerHosts kafkaHosts = new ZkHosts(kafkaHost);
        SpoutConfig kafkaConfig = new SpoutConfig(kafkaHosts, topicName, "/kafka", topicName);
        kafkaConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
        KafkaSpout kafkaSpout = new KafkaSpout(kafkaConfig);
        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("twitter-spout", kafkaSpout, 8);
        builder.setBolt("WordSplitterBolt", new JsonWordSplitterBolt(5)).shuffleGrouping("twitter-spout");
        builder.setBolt("IgnoreWordsBolt", new IgnoreWordsBolt()).shuffleGrouping("WordSplitterBolt");
        builder.setBolt("WordCounterBolt", new WordCounterBolt(5, 5 * 60, 50)).shuffleGrouping("IgnoreWordsBolt");
        Config config = new Config();
        config.setDebug(false);
        config.setMaxTaskParallelism(5);
        config.put(Config.NIMBUS_HOST, nimbusHost);
        config.put(Config.NIMBUS_THRIFT_PORT, 6627);
        config.put(Config.STORM_ZOOKEEPER_PORT, 2181);
        config.put(Config.STORM_ZOOKEEPER_SERVERS, Arrays.asList(kafkaIp));
        try {
                config.setNumWorkers(20);
                config.setMaxSpoutPending(5000);
                StormSubmitter.submitTopology(topologyName, config, builder.createTopology()); 
            } catch (Exception e) {
                    throw new IllegalStateException("Couldn't initialize the topology", e);
            }
    }
}

我在Storm UI中遇到此异常：

Unable to get offset lags for kafka. Reason: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/twitter1/partitions at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:242) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:231) at org.apache.curator.shaded.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) at org.apache.curator.shaded.RetryLoop.callWithRetry(RetryLoop.java:100) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:228) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:219) at org.apache.curator.shaded.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:41) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getLeadersAndTopicPartitions(KafkaOffsetLagUtil.java:319) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.getOffsetLags(KafkaOffsetLagUtil.java:256) at org.apache.storm.kafka.monitor.KafkaOffsetLagUtil.main(KafkaOffsetLagUtil.java:124)

错误Unable to get offset lags for kafka保持不变，而异常的其他部分根据我更改的zkroot路径（SpoutConfig中的第3个参数）而变化。我不知道如何填写这些论点，让Kafka从我的主题中获取推文。

我使用此处提供的教程编写用于提交拓扑的代码：http://stdatalabs.blogspot.ca/2016/10/real-time-stream-processing-using.html 我已经为maven依赖项做了很多更改。我的pom.xml包含了storm-core，kafka等的所有依赖项，以及maven repo中提供的最新版本。

Answer 1

zkHosts() 应该包含 zookeeper 的配置而不是 kafka。如果你的 zookeeper 和 kafka 在同一台服务器上。

尝试为zookeeper(2181)提供正确的端口

参考https://storm.apache.org/releases/1.2.3/storm-kafka.html

风暴/卡夫卡 - 无法获得卡夫卡的偏移滞后

1 个答案: