Apache Storm：storm-kafka-monitor脚本抛出异常

时间：2018-06-05 07:33:39

标签： java apache-kafka apache-storm apache-storm-topology

我已在本地设置了Apache Storm群集并提交了一个拓扑，该拓扑在Storm UI中可见。一切准备就绪后，我在ui.log文件中收到以下异常。该错误与storm-kafka-monitor有点相关，这是一个bash脚本，我在windows上运行它 任何帮助将受到高度赞赏。在此先感谢：）

2018-06-05 11:26:20.293 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception thrown while getting lag for spout id: kafka-spout and spout class: org.apache.storm.kafka.KafkaSpout
2018-06-05 11:26:20.294 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception message:Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
java.io.IOException: Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_162]
    at org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:180) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.getLagResultForKafka(TopologySpoutLag.java:163) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.getLagResultForOldKafkaSpout(TopologySpoutLag.java:193) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.lag(TopologySpoutLag.java:59) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$topology_lag.invoke(core.clj:805) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$fn__9586.invoke(core.clj:1165) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$make_route$fn__5979.invoke(core.clj:100) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$if_route$fn__5967.invoke(core.clj:46) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$if_method$fn__5960.invoke(core.clj:31) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$routing$fn__5985.invoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
    at clojure.core$some.invoke(core.clj:2570) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
    at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[clojure-1.7.0.jar:?]
    at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.shade.compojure.core$routes$fn__5989.invoke(core.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__8894.invoke(cors.clj:149) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8841.invoke(json.clj:56) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7904.invoke(reload.clj:22) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.helpers$requests_middleware$fn__6874.invoke(helpers.clj:52) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$catch_errors$fn__9761.invoke(core.clj:1428) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__6541.invoke(keyword_params.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__6584.invoke(nested_params.clj:84) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__6513.invoke(params.clj:64) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__6836.invoke(flash.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__6822.invoke(session.clj:98) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__6371.invoke(servlet.clj:127) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet$servlet$fn__6375.invoke(servlet.clj:136) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
    at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__6967.invoke(helpers.clj:192) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) ~[storm-core-1.2.1.jar:1.2.1]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
    at java.lang.ProcessImpl.create(Native Method) ~[?:1.8.0_162]
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) ~[?:1.8.0_162]
    at java.lang.ProcessImpl.start(ProcessImpl.java:137) ~[?:1.8.0_162]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_162]
    ... 68 more

1 个答案:

答案 0 :(得分：1)

正如您所知，脚本是一个bash文件。该功能（显示日志末端在Storm UI中偏移了喷口的距离）目前在Windows上不起作用。这有点不方便，但它不应该阻止你的拓扑正常工作。

如果你想知道spout的日志末端偏移量有多远，你可以在你的一台Kafka机器上使用kafka-consumer-groups.bat脚本。如果您使用的是storm-kafka-client喷口，则喷口延迟也是该组件提供的指标的一部分。

我希望我们可以在某些时候摆脱storm-kafka-monitor（它有一些其他问题https://issues.apache.org/jira/browse/STORM-2600），但在此之前，如果你想尝试提供Windows兼容性，那么你' d需要将https://github.com/apache/storm/blob/master/bin/storm-kafka-monitor移植到cmd / powershell，并修改https://github.com/apache/storm/blob/4137328b75c06771f84414c3c2113e2d1c757c08/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L146，以便根据操作系统调用Windows或bash脚本。

相关问题
最新问题