apache风暴喷口没有散发出元组

时间：2015-12-08 03:09:34

标签： amazon-web-services apache-storm

我已使用三台计算机和this指南在aws群集上设置了测试环境。

我在本地模式下测试了我的代码，并使用wirbelsturm创建了一个本地流浪群集，这两个群集都可以提供所需的结果。

当我现在将我的代码提交到网络服务器时，我的鲸鱼喷水和我的所有螺栓都是沉默的。我的鲸鱼嘴从csv读取，我已经复制到了灵气和我的主管。风暴UI显示拓扑为活动状态并显示所有螺栓和我的喷口，但计数器不可见。主管没有二手工人。防火墙配置为让nimbus和supervisor接受来自supervisor和nimbus的端口6700-6703。动物园管理员是否在那些港口谈话？

我似乎无法在我的机器上找到输出日志。我发现ui和nimbus登录了/ usr / local / storm / logs的nimbus和slave，但除此之外我似乎没有收到错误甚至是针对spouts / bolt的日志。 vagrant机器显示一个worker-xxxx.log，但我的aws服务器没有。

是因为我的代码因某些错误而崩溃，还是因为我的配置错误了？

更新：我使用风暴启动示例验证了我的拓扑，这些似乎也没有用。我使用mvn包来构建一个uberjar。

UPDATE2：

我从我的主管那里收录了日志，并没有显示任何错误，但也许还有一些东西...... 

 2015-12-08 13:42:55.168 b.s.u.Utils [INFO] Using defaults.yaml from resources
2015-12-08 13:42:55.297 b.s.u.Utils [INFO] Using storm.yaml from resources
2015-12-08 13:42:57.434 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:host.name=ip-172-31-26-239.us-west-2.compute.internal
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_91
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/local/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/usr/local/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/usr/local/apache-sto$
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp
2015-12-08 13:42:57.435 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:java.compiler=<NA>
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-504.8.1.el6.x86_64
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.name=storm
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.home=/app/home/storm
2015-12-08 13:42:57.436 o.a.s.s.o.a.z.ZooKeeper [INFO] Client environment:user.dir=/
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:host.name=ip-172-31-26-239.us-west-2.compute.internal
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.version=1.7.0_91
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.vendor=Oracle Corporation
2015-12-08 13:42:57.459 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.class.path=/usr/local/apache-storm-0.10.0/lib/clojure-1.6.0.jar:/usr/local/apache-storm-0.10.0/lib/log4j-core-2.1.jar:/usr/local/apache-storm-0.10.0/lib/log4j-api-2.1.jar:/usr/local/ap$
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.io.tmpdir=/tmp
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:java.compiler=<NA>
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.name=Linux
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.arch=amd64
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:os.version=2.6.32-504.8.1.el6.x86_64
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.name=storm
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.home=/app/home/storm
2015-12-08 13:42:57.460 o.a.s.s.o.a.z.s.ZooKeeperServer [INFO] Server environment:user.dir=/
2015-12-08 13:42:57.774 b.s.u.Utils [INFO] Using defaults.yaml from resources
2015-12-08 13:42:57.803 b.s.u.Utils [INFO] Using storm.yaml from resources
2015-12-08 13:42:57.939 b.s.d.supervisor [INFO] Starting Supervisor with conf {"topology.builtin.metrics.bucket.size.secs" 60, "nimbus.childopts" "-Xmx1024m -Djava.net.preferIPv4Stack=true", "ui.filter.params" nil, "storm.cluster.mode" "distributed", "storm.messaging.net$
2015-12-08 13:42:57.963 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-12-08 13:42:58.063 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-12-08 13:42:58.066 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181 sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@502016b8
2015-12-08 13:42:58.081 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server zkServer1/xx.xx.xx.xx:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-08 13:42:58.089 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to zkServer1/xx.xx.xx.xx:2181, initiating session
2015-12-08 13:42:58.094 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server zkServer1/xx.xx.xx.xx:2181, sessionid = 0x15182c7ba25000d, negotiated timeout = 20000
2015-12-08 13:42:58.096 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-12-08 13:42:58.097 b.s.zookeeper [INFO] Zookeeper state update: :connected:none
2015-12-08 13:42:59.109 o.a.s.s.o.a.z.ClientCnxn [INFO] EventThread shut down
2015-12-08 13:42:59.110 o.a.s.s.o.a.z.ZooKeeper [INFO] Session: 0x15182c7ba25000d closed
2015-12-08 13:42:59.111 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] the maxRetries [5]
2015-12-08 13:42:59.116 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2015-12-08 13:42:59.116 o.a.s.s.o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=zkserver1:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@5edfa0aa
2015-12-08 13:42:59.121 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server zkServer1/xx.xx.xx.xx:2181. Will not attempt to authenticate using SASL (unknown error)
2015-12-08 13:42:59.122 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection established to zkServer1/xx.xx.xx.xx:2181, initiating session
2015-12-08 13:42:59.124 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server zkServer1/xx.xx.xx.xx:2181, sessionid = 0x15182c7ba25000e, negotiated timeout = 20000
2015-12-08 13:42:59.124 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2015-12-08 13:42:59.169 b.s.d.supervisor [INFO] Starting supervisor with id cc5e1723-cc06-4bc1-a1bf-192a1d7f5bf6 at host xxxxxxx.us-west-2.compute.internal
2015-12-08 13:43:06.059 b.s.d.supervisor [INFO] Downloading code for storm id production-topology-4-1449599549 from /app/storm/nimbus/stormdist/production-topology-4-1449599549
2015-12-08 13:43:06.075 b.s.u.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [2000] the maxSleepTimeMs [60000] the maxRetries [5]

有什么想法吗？

UPDATE2：

所以我确实找到了这个：

java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out
        at backtype.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:59) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:51) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:103) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:72) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:74) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:37) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.Utils.downloadFromMaster(Utils.java:361) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.daemon.supervisor$fn__7720.invoke(supervisor.clj:581) ~[storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.MultiFn.invoke(MultiFn.java:241) ~[clojure-1.6.0.jar:?]
        at backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__7638.invoke(supervisor.clj:465) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:40) [storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_91]
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection timed out
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:187) ~[storm-core-0.10.0.jar:0.10.0]
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) ~[storm-core-0.10.0.jar:0.10.0]
        ... 11 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.7.0_91]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~[?:1.7.0_91]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~[?:1.7.0_91]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~[?:1.7.0_91]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.7.0_91]
        at java.net.Socket.connect(Socket.java:579) ~[?:1.7.0_91]
        at org.apache.thrift7.transport.TSocket.open(TSocket.java:182) ~[storm-core-0.10.0.jar:0.10.0]
        at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:103) ~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:48) ~[storm-core-0.10.0.jar:0.10.0]
        ... 11 more
2015-12-08 14:26:41.028 b.s.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336) [storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
        at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:48) [storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_91]

1 个答案:

答案 0 :(得分：1)

我按照与你相同的指南，遇到了同样的问题。 是什么解决了我的问题：

以相同的方式编辑三台机器（zookeeper，nimbus和slave1）的3个/etc/hosts文件

  • 首先删除以::1开头的IPv6线路，apache storm不支持​​此功能。
  • 在包含本地别名的文件的第一行中，在127.0.0.1之后放置本地计算机的公共主机名（群集的其他节点已知的主机名）。我想这是别名风暴将考虑在内。
  • 最后，正如指南列表中所说的所有其他机器以及风暴知识的主机名

最后我的/etc/hosts看起来像这样（对于雨云）

127.0.0.1 vm-matthias-02 localhost.localdomain localhost
192.168.200.48  vm-matthias-01
192.168.200.49  vm-matthias-02
192.168.200.50  vm-matthias-03

在编辑配置文件时，请注意使用相同的机器名称。

相关问题
最新问题