Docker与Storm 0.10.0和Zookeeper（连接丢失）组成

时间：2016-03-31 14:13:33

标签： node.js docker apache-kafka apache-storm apache-zookeeper

部署了以下架构：

  1. Docker容器中的Apache Kafka和Zookeeper：https://hub.docker.com/r/eucm/kzk/
  2. 使用这些图像的Storm（Nimbus，Supervisor，UI）：

    3. 使用以下'docker-compose.yml'

    # A scalable cache
#     saves data to ./data/mongo
mongo:
    image: mongo:3.0
    container_name: "mongo"
    volumes:
      - "./data/mongo:/data/db"
    ports:
      - "27017:27017"

# Kafka (A scalable queue) + Zookeeper (Distributed conf. service)
# as of 2015.11 -- kafka:0.8.2.1, zookeeper:3.4.5+dfsg-2
kzk:
    image: eucm/kzk
    container_name: "kzk"
    volumes:
      - "./data/zookeeper:/var/lib/zookeeper"
      - "./data/kafka:/tmp/kafka-logs"
    environment:
      - ADVERTISED_HOST=localhost
      - AUTO_CREATE_TOPICS=true
      - DELETE_TOPICS=true
    ports:
      - "9092:9092"
      - "2181:2181"

# nimbus: supervises everything; talks to zookeeper
# in e-ucm/dockerized-storm -- storm:0.10.0
#    requires: kzk
nimbus:
    image: eucm/storm-nimbus
    container_name: "nimbus"
    environment:
      - KZK_PORT_2181_TCP_ADDR=kzk
      - NIMBUS_PORT_6627_TCP_ADDR=nimbus
      - UI_PORT_8081_TCP_ADDR=ui
    ports:
      - "6627:6627"

# ui: allows access to logs
# in e-ucm/dockerized-storm -- storm:0.10.0
#    requires: nimbus, kzk
ui:
    image: eucm/storm-ui
    container_name: "ui"
    environment:
      - KZK_PORT_2181_TCP_ADDR=kzk
      - NIMBUS_PORT_6627_TCP_ADDR=nimbus
      - UI_PORT_8081_TCP_ADDR=ui
    ports: 
        - "8081:8081"

# supervisor: supervises actual workers
# in e-ucm/dockerized-storm -- storm:0.10.0
#   requires: nimbus, kzk, mongo
supervisor:
    image: eucm/storm-supervisor
    container_name: "supervisor"
    environment:
      - KZK_PORT_2181_TCP_ADDR=kzk
      - NIMBUS_PORT_6627_TCP_ADDR=nimbus
      - UI_PORT_8081_TCP_ADDR=ui
    ports:
      - "6700:6700"
      - "6701:6701"
      - "6702:6702"
      - "6703:6703"

    我们还使用NodeJS和kafka-node包（https://github.com/SOHU-Co/kafka-node）来创建一个名为'sessionTest'的主题。

    我将这个拓扑（https://github.com/e-ucm/rage-analytics-realtime/blob/master/src/main/java/es/eucm/gleaner/realtime/RealTime.java）从主机发送到Storm。使用以下命令：

    /home/dan/storm-test/storm/apache-storm-0.10.0/bin/storm jar /home/dan/storm-test/realtime-jar-with-dependencies.jar es.eucm.gleaner.realtime.RealTime sessionTest mongodb://localhost:27017/analytics-backend-test localhost:2181 -c nimbus.host=localhost

    过了一会儿，看着Storm工作人员记录（在主管图像内），我们看到以下错误。

    2016-03-30 09:30:05.984 b.s.d.worker [INFO] Shutting down worker sessionTest-3-1459329789 2c94d927-2341-4649-bee6-b0bb13f2706f 6700
2016-03-30 09:30:05.985 b.s.d.worker [INFO] Shutting down receive thread
2016-03-30 09:30:06.006 b.s.m.n.Client [INFO] creating Netty Client, connecting to supervisor:6700, bufferSize: 5242880
2016-03-30 09:30:06.009 b.s.m.loader [INFO] Shutting down receiving-thread: [sessionTest-3-1459329789, 6700]
2016-03-30 09:30:06.009 b.s.m.n.Client [INFO] closing Netty Client Netty-Client-supervisor/172.20.0.5:6700
2016-03-30 09:30:06.009 b.s.m.n.Client [INFO] waiting up to 600000 ms to send 0 pending messages to Netty-Client-supervisor/172.20.0.5:6700
2016-03-30 09:30:06.009 b.s.m.loader [INFO] Waiting for receiving-thread:[sessionTest-3-1459329789, 6700] to die
2016-03-30 09:30:06.010 b.s.m.loader [INFO] Shutdown receiving-thread: [sessionTest-3-1459329789, 6700]
2016-03-30 09:30:06.010 b.s.d.worker [INFO] Shut down receive thread
2016-03-30 09:30:06.010 b.s.d.worker [INFO] Terminating messaging context
2016-03-30 09:30:06.010 b.s.d.worker [INFO] Shutting down executors
2016-03-30 09:30:06.011 b.s.d.executor [INFO] Shutting down executor $spoutcoord-spout0:[2 2]
2016-03-30 09:30:06.011 b.s.util [INFO] Async loop interrupted!
2016-03-30 09:30:06.789 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) ~[stormjar.jar:?]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[stormjar.jar:?]
    at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[stormjar.jar:?]
    at storm.kafka.trident.Coordinator.<init>(Coordinator.java:33) ~[stormjar.jar:?]http://e-ucm.github.io/a2/#api-Login-Login
    at storm.kafka.trident.OpaqueTridentKafkaSpout.getCoordinator(OpaqueTridentKafkaSpout.java:46) ~[stormjar.jar:?]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Coordinator.<init>(OpaquePartitionedTridentSpoutExecutor.java:44) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor.getCoordinator(OpaquePartitionedTridentSpoutExecutor.java:183) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.topology.MasterBatchCoordinator.open(MasterBatchCoordinator.java:113) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.daemon.executor$fn__5624$fn__5639.invoke(executor.clj:564) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal]
Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[stormjar.jar:?]
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[stormjar.jar:?]
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[stormjar.jar:?]
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
2016-03-30 09:30:06.794 b.s.d.executor [ERROR] 
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:81) ~[stormjar.jar:?]
    at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:42) ~[stormjar.jar:?]
    at storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:57) ~[stormjar.jar:?]
    at storm.kafka.trident.Coordinator.<init>(Coordinator.java:33) ~[stormjar.jar:?]
    at storm.kafka.trident.OpaqueTridentKafkaSpout.getCoordinator(OpaqueTridentKafkaSpout.java:46) ~[stormjar.jar:?]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Coordinator.<init>(OpaquePartitionedTridentSpoutExecutor.java:44) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.spout.OpaquePartitionedTridentSpoutExecutor.getCoordinator(OpaquePartitionedTridentSpoutExecutor.java:183) ~[storm-core-0.10.0.jar:0.10.0]
    at storm.trident.topology.MasterBatchCoordinator.open(MasterBatchCoordinator.java:113) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.daemon.executor$fn__5624$fn__5639.invoke(executor.clj:564) ~[storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal]
Caused by: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:94) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/topics/sessionTest/partitions
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[stormjar.jar:?]
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[stormjar.jar:?]
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:203) ~[stormjar.jar:?]
    at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[stormjar.jar:?]
    at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[stormjar.jar:?]
    at storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:65) ~[stormjar.jar:?]
    ... 11 more
2016-03-30 09:30:06.805 b.s.util [ERROR] Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
    at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
    at backtype.storm.daemon.worker$fn__7188$fn__7189.invoke(worker.clj:536) [storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.daemon.executor$mk_executor_data$fn__5523$fn__5524.invoke(executor.clj:261) [storm-core-0.10.0.jar:0.10.0]
    at backtype.storm.util$async_loop$fn__545.invoke(util.clj:489) [storm-core-0.10.0.jar:0.10.0]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal]

    我已经确认我可以将数据发送到kafka docker镜像并正确使用它们（通过控制台消费者）。

    TLDR：部署了Kafka，Zookeeper，Storm作为docker图像。从主机向kafka发送数据，风暴工作者（应该从kafka使用）显示与zookeeper的连接丢失日志。

    任何想法都会有用，而不是x！

0 个答案:

没有答案
相关问题
最新问题