Datastax Enterprise:nodetool清理期间出错

时间:2014-08-27 20:49:22

标签: datastax-enterprise datastax

我有一个带有cassandra和搜索节点的双数据中心部署。它看起来像这样:

ubuntu@ip-172-31-25-223:~$ dsetool ring
Note: Ownership information does not include topology, please specify a keyspace. 
Address          DC           Rack         Workload         Status  State    Load             Owns                 VNodes                                      
172.31.47.194    Solr         2a           Unknown          Up      Normal   3.17 GB          0.00%                1                                           
172.31.39.59     us-west-2    2a           Unknown          Up      Normal   2.32 GB          31.53%               512                                         
172.31.9.36      us-west-2    2c           Unknown          Up      Normal   3.43 GB          33.48%               512                                         
172.31.25.223    us-west-2    2b           Unknown          Up      Normal   3.25 GB          34.99%               512                                         
Warning:  Node 172.31.25.223 is serving 1.11 times the token space of node 172.31.39.59, which means it will be using 1.11 times more disk space and network bandwidth. If this is unintentional, check out http://wiki.apache.org/cassandra/Operations#Ring_management

我最近添加了172.31.39.59节点,我想在其他节点上运行清理以删除一些重复数据。

nodetool cleanup节点上运行172.31.25.223会产生以下跟踪:

ubuntu@ip-172-31-25-223:~$ nodetool cleanup
Error occurred during cleanup
java.util.concurrent.ExecutionException: java.lang.IllegalStateException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:228)
        at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:266)
        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1112)
        at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2251)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException
        at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex.delete(Cql3SolrSecondaryIndex.java:61)
        at org.apache.cassandra.db.index.SecondaryIndexManager.deleteFromIndexes(SecondaryIndexManager.java:470)
        at org.apache.cassandra.db.compaction.CompactionManager$CleanupStrategy$Full.cleanup(CompactionManager.java:720)
        at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:580)
        at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:63)
        at org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:275)
        at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:223)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        ... 3 more
Caused by: java.lang.NullPointerException
        at com.datastax.bdp.search.solr.AbstractSolrSecondaryIndex.doDelete(AbstractSolrSecondaryIndex.java:619)
        at com.datastax.bdp.search.solr.Cql3SolrSecondaryIndex.delete(Cql3SolrSecondaryIndex.java:57)
        ... 10 more

我是Datastax企业的新手,我对Cassandra-> Solr以及管道工作方式的了解不多。我很感激任何帮助!

谢谢!

Advait

更新:尝试再次运行它并且这次得到了不同的跟踪:

ubuntu@ip-172-31-25-223:~$ nodetool cleanup
Exception in thread "main" java.lang.AssertionError: [SSTableReader(path='/raid0/cassandra/data/liminex_ent/sub_accounts/liminex_ent-sub_accounts-jb-73-Data.db'), SSTableReader(path='/raid0/cassandra/data/liminex_ent/sub_accounts/liminex_ent-sub_accounts-jb-74-Data.db')]
        at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2130)
        at org.apache.cassandra.db.ColumnFamilyStore$13.call(ColumnFamilyStore.java:2127)
        at org.apache.cassandra.db.ColumnFamilyStore.runWithCompactionsDisabled(ColumnFamilyStore.java:2109)
        at org.apache.cassandra.db.ColumnFamilyStore.markAllCompacting(ColumnFamilyStore.java:2140)
        at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:215)
        at org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:266)
        at org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:1112)
        at org.apache.cassandra.service.StorageService.forceKeyspaceCleanup(StorageService.java:2251)
        at sun.reflect.GeneratedMethodAccessor105.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at sun.rmi.transport.Transport$1.run(Transport.java:174)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

1 个答案:

答案 0 :(得分:1)

这是一个错误 - 在4.5.3

中修复

清理错误的修复程序在我们最新发布的4.5.3版本中,即2014年11月4日的GA。

滚动升级应该是直接的/没有停机时间。

这是发行说明:

http://www.datastax.com/documentation/datastax_enterprise/4.5/datastax_enterprise/RNdse45.html

  

修复了在非Solr工作负载上导致空指针异常的问题   节点持有Solr数据并尝试运行nodetool清理   对数据的命令。 (DSP-4310)