Question

我有一个小型（5节点）群集，我用它来对DSE搜索的性能进行基准测试，并使用各种特定的索引策略（我在CentOS上使用DSE 4.5.2）。当我一次只提交一个查询时，一切正常，但当我增加负载时，我很快就会看到netty异常。在SolrJ中（我的大部分测试都使用了java客户端），这些显示为

Error with server [MY URL]: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: address already in use by: [id: 0x2d3873c4, local:E:2d3873c4 => local:local]

每当我有多个线程同时查询集群时，无论我实际指向哪个节点，无论请求是否来自同一服务器，这似乎都会一直发生。在使用其他客户端（例如PySolr4）时，我也能够引发这种行为，所以这似乎是一个DSE设置问题，而不是我的测试应用程序的一些奇怪问题。

此外，如果我将dse.yaml中的节点间通信类型从netty更改为http，则同一群集将很乐意响应提交数百个并发请求的客户端。我最初在dse.yaml中保留了默认的netty *设置，但在遇到错误后，我玩了几个选项（max_connections，acceptor / worker threads等）;不幸的是，我没有找到关于这些值的大量文档，我改变的任何内容似乎都没有达到预期的效果。

从概念上讲，似乎使用netty会提供比http更好的性能，所以我想试着弄清楚这里可能出现的问题。提前感谢任何建议。

请参阅以下/var/log/cassandra/system.log对应于其中一个错误的堆栈跟踪：

ERROR [http-8983-exec-4] 2014-11-19 18:35:51,557 SolrDispatchFilter.java (line 696) Error request exception: address already in use by: [id: 0x17617168, local:E:17617168 => local:local]
java.lang.RuntimeException: address already in use by: [id: 0x17617168, local:E:17617168 => local:local]
    at com.datastax.bdp.search.solr.handler.shard.netty.ShardClient.tryNewConnection(ShardClient.java:204)
    at com.datastax.bdp.search.solr.handler.shard.netty.ShardClient.sendTo(ShardClient.java:163)
    at com.datastax.bdp.search.solr.handler.shard.netty.NettyShardHandler.submit(NettyShardHandler.java:76)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:287)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:137)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1889)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:723)
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:185)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
    at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:147)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:218)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:100)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
    at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:891)
    at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:750)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2283)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: io.netty.channel.ChannelException: address already in use by: [id: 0x17617168, local:E:17617168 => local:local]
    at io.netty.channel.local.LocalChannelRegistry.register(LocalChannelRegistry.java:46)
    at io.netty.channel.local.LocalChannel.doBind(LocalChannel.java:178)
    at io.netty.channel.local.LocalChannel$LocalUnsafe.connect(LocalChannel.java:349)
    at io.netty.channel.DefaultChannelPipeline$HeadHandler.connect(DefaultChannelPipeline.java:1008)
    at io.netty.channel.DefaultChannelHandlerContext.invokeConnect(DefaultChannelHandlerContext.java:495)
    at io.netty.channel.DefaultChannelHandlerContext.connect(DefaultChannelHandlerContext.java:480)
    at io.netty.channel.DefaultChannelHandlerContext.connect(DefaultChannelHandlerContext.java:465)
    at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:847)
    at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:199)
    at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:165)
    at io.netty.channel.local.LocalEventLoop.run(LocalEventLoop.java:33)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
    ... 1 more

Answer 1

似乎是与https://github.com/netty/netty/issues/1765相关的Netty / JVM错误。你能尝试另一个操作系统和/或JVM版本吗？

DSE Solr：并发查询请求期间的Netty错误

1 个答案: