Question

我早上检查了Cygnus日志并查看了这些频道错误。似乎ckan-channel已满，无法继续交易。

您对此错误有什么建议？

16/06/01 08:11:58 WARN http.HTTPSource: Error appending event to channel. Channel might be full. Consider increasing the channel capacity or make sure the sinks perform faster.
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: ckan-channel}
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
        at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 16 mor16/06/01 08:11:58 WARN http.HTTPSource: Error appending event to channel. Channel might be full. Consider increasing the channel capacity or make sure the sinks perform faster.
org.apache.flume.ChannelException: Unable to put batch on required channel: org.apache.flume.channel.MemoryChannel{name: ckan-channel}
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
        at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:201)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
        at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:128)
        at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
        at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
        ... 16 more

此外，我发现hfds-channel / sink在日志中甚至没有活动。

这是因为ckan-channel出错了所以没有调用hdfs-channel吗？

Answer 1

您遇到的是来自Flume的众所周知的日志错误。该日志本身就是描述性的：内部通信源和接收器的其中一个通道正在充满。

原因？源生成的事件比接收器能够处理的事件更快。在这种情况下，有几件事情可以做：

减少发送给Cygnus的通知量。这看起来很明显，但你可能会说“这是我的数据，我想坚持下去”。好吧，我们正在谈论减少通知的数量，而不是减少数据量。例如，不是发送关于某个实体的每个属性的通知已发生变化，而是在同一通知中发送所有属性。这可以通过改变您订阅Cygnus到Orion的方式来实现。减少通知数量的原因是信道容量是根据Flume事件测量的（通常，每个通知有一个Flume事件），而不是以字节为单位。因此，拥有更大的通知将允许您保留相同的数据，但消耗更少的通道容量。
增加频道的容量。这仅在通知吞吐量不规则时才有用。如果经常，这只会延迟问题。
使用批处理。这在所有支持此功能的接收器中得到了广泛的解释（顺便说一句，大多数都是这样）。例如，看看有关HDFS汇documentation的解释，但所有汇点的结论都相同。
使用并行化。不要使用单个接收器，而是尝试添加更多并行运行的接收器。这是通过将所有这些接收器连接到同一个通道来实现的，但恕我直言，最好的是每个接收器都有一个通道并使用我们的自定义RoundRobinChannelSelector。这是一个非常先进的功能，所以请问我以前的功能是否适合你。

您可以查看Performance section。

fiware天鹅座频道已满

1 个答案: