如何将kafka 0.7.2升级到kafka 0.8？

时间：2014-08-02 07:28:27

标签： apache-kafka apache-storm

我正在使用storm和kafka进行实时网络分析。

目前我正在使用以下版本，它运行正常。

注意：

Storm - 0.9.0 
kafka - 0.7.2 
zookeeper - 3.3.3

我想upgrade到kafka 0.8 stable version。

问题：

很难找到与风暴兼容的最新版kafka的依赖项。

kafka 0.8稳定版有风暴吗？

任何参考或建议都将不胜感激。

/ ************* ************************************************** *************** /

1.我参考了答案Enno Shioji

尝试了这个项目https://github.com/wurstmeister/storm-kafka-0.8-plus-test

我收到以下error 

java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/clickstreamlog/partitions
at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.KafkaSpout.open(KafkaSpout.java:87) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$fn__3284$fn__3299.invoke(executor.clj:520) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:429) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:679) [na:1.6.0_27]
Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/clickstreamlog/partitions
at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
... 7 common frames omitted
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/clickstreamlog/partitions
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1586) ~[zookeeper-3.4.5.jar:3.4.5-1392090]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[curator-framework-2.4.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[curator-framework-2.4.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.4.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[curator-framework-2.4.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[curator-framework-2.4.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[curator-framework-2.4.0.jar:na]
at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[storm-kafka-0.9.2-incubating.jar:0.9.2-incubating]
... 8 common frames omitted

拓扑

package storm;

import java.io.FileNotFoundException;
import java.io.IOException;

import storm.kafka.BrokerHosts;
import storm.kafka.KafkaSpout;
import storm.kafka.SpoutConfig;
import storm.kafka.ZkHosts;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseBasicBolt;
import backtype.storm.tuple.Tuple;
import backtype.storm.utils.Utils;

public class TopologyMain {

public static void main(String[] args) throws FileNotFoundException,
        IOException {

    Config config = new Config();
    config.setDebug(true);

    TopologyBuilder topology = new TopologyBuilder();

    BrokerHosts hosts = new ZkHosts("127.0.0.1");
    SpoutConfig kafkaSpoutConfig = new SpoutConfig(hosts,"clickstreamlog", "127.0.0.1","discovery");

    KafkaSpout spout = new KafkaSpout(kafkaSpoutConfig);
    topology.setSpout("spout", spout,4);
    topology.setBolt("print", new PrinterBolt());

    LocalCluster cluster = new LocalCluster();
    cluster.submitTopology("Log", config, topology.createTopology());
    Utils.sleep(10000);
    cluster.killTopology("Log");
}

public static class PrinterBolt extends BaseBasicBolt {

    private static final long serialVersionUID = 1L;

    public void declareOutputFields(OutputFieldsDeclarer declarer) {

    }

    public void execute(Tuple tuple, BasicOutputCollector collector) {
         System.out.println(tuple.toString());
    }

}

}

Here pom.xml文件。

2.我也曾尝试过这个项目https://github.com/eshioji/trident-tutorial。

Note:我使用的是具有不同依赖关系{（3}}

的相同拓扑代码

我收到以下error 

java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method) ~[na:1.6.0_27]
at java.util.zip.ZipFile.<init>(ZipFile.java:132) ~[na:1.6.0_27]
at java.util.zip.ZipFile.<init>(ZipFile.java:93) ~[na:1.6.0_27]
at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source) ~[na:na]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.6.0_27]
at java.lang.reflect.Constructor.newInstance(Constructor.java:532) ~[na:1.6.0_27]
at clojure.lang.Reflector.invokeConstructor(Reflector.java:180) ~[clojure-1.5.1.jar:na]
at backtype.storm.util$zip_contains_dir_QMARK_.invoke(util.clj:777) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$resources_jar$fn__5016.invoke(supervisor.clj:506) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.core$filter$fn__4226.invoke(core.clj:2533) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:67) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.first(LazySeq.java:82) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.first(RT.java:577) ~[clojure-1.5.1.jar:na]
at clojure.core$first.invoke(core.clj:55) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.supervisor$resources_jar.invoke(supervisor.clj:504) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.supervisor$fn__5021.invoke(supervisor.clj:514) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__4917.invoke(supervisor.clj:325) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.event$event_manager$fn__2379.invoke(event.clj:39) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:679) [na:1.6.0_27]

1 个答案:

答案 0 :(得分：0)

有一个使用docker：
https://github.com/wurstmeister/storm-kafka-0.8-plus-test

如果你更喜欢Vagrant我有一个使用0.9.2＆amp;的回购。 0.8.4：
https://github.com/eshioji/trident-tutorial

相关问题
最新问题