这是一个看似相关问题的相当大的帖子，我们也发布到了Cassandra用户列表中。我们正在使用Cassandra 3.3，我们的开发一直试图找出我们在我们的一个预制环境中看到的一些新问题，我们经常遇到一致的故障（每天或每2-3个即使负载/交易数量很少，也可以。） 我们正在运行2个数据中心部署，每个数据中心有3个节点。我们的表设置为复制因子= 2，我们有16G专用于堆，G1GC用于垃圾收集。我们的系统是AWS M4.2xlarge，具有8个CPU和32GB RAM，每个节点上有2个通用EBS卷，每个节点500GB。 一旦我们点击它，似乎恢复的唯一方法是关闭群集并重新启动。重新启动后运行修复通常会导致失败，我们最终不得不在再次启动清理之前截断表。我们不确定这两者是否相互关联。我们几乎在所有节点上都看到了同样的问题。
如果有人对如何进一步诊断/调试有任何提示或建议，那将会有很大帮助！问题是：
问题1：一旦出现错误，他们只会重复一点，然后是问题2中的错误。
INFO [CompactionExecutor:165] 2017-01-08 08:32:39,915 AutoSavingCache.java:386 - Saved KeyCache (63 items) in 5 ms
INFO [IndexSummaryManager:1] 2017-01-08 08:32:41,438 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [HANDSHAKE-ahldataslave4.bos.manhattan.aspect-cloud.net/10.184.8.224] 2017-01-08 09:30:03,988 OutboundTcpConnection.java:505 - Handshaking version with ahldataslave4.bos.manhattan.aspect-cloud.net/10.184.8.224
INFO [IndexSummaryManager:1] 2017-01-08 09:32:41,440 IndexSummaryRedistribution.java:74 - Redistributing index summaries
WARN [SharedPool-Worker-9] 2017-01-08 10:30:00,116 BatchStatement.java:289 - Batch of prepared statements for [manhattan.rcmessages] is of size 9264, exceeding specified threshold of 5120 by 4144.
INFO [IndexSummaryManager:1] 2017-01-08 10:32:41,442 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 11:32:41,443 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [CompactionExecutor:162] 2017-01-08 12:32:39,914 AutoSavingCache.java:386 - Saved KeyCache (108 items) in 4 ms
INFO [IndexSummaryManager:1] 2017-01-08 12:32:41,446 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 13:32:41,448 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 14:32:41,450 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [IndexSummaryManager:1] 2017-01-08 15:32:41,451 IndexSummaryRedistribution.java:74 - Redistributing index summaries
INFO [CompactionExecutor:170] 2017-01-08 16:32:39,915 AutoSavingCache.java:386 - Saved KeyCache (109 items) in 4 ms
INFO [IndexSummaryManager:1] 2017-01-08 16:32:41,453 IndexSummaryRedistribution.java:74 - Redistributing index summaries
WARN [SharedPool-Worker-4] 2017-01-08 17:30:45,048 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-4,5,main]: {}
java.lang.AssertionError: null
at org.apache.cassandra.db.rows.BufferCell.<init>(BufferCell.java:49) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:88) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.tombstone(BufferCell.java:83) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BufferCell.purge(BufferCell.java:175) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.ComplexColumnData.lambda$purge$107(ComplexColumnData.java:165) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.ComplexColumnData.transformAndFilter(ComplexColumnData.java:170) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:165) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.ComplexColumnData.purge(ComplexColumnData.java:43) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.lambda$purge$102(BTreeRow.java:333) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree$FiltrationTracker.apply(BTree.java:650) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:693) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.btree.BTree.transformAndFilter(BTree.java:668) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.transformAndFilter(BTreeRow.java:338) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:333) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:116) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:133) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:79) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:134) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:127) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:123) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:292) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:50) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
问题2：
WARN [SharedPool-Worker-2] 2017-01-09 05:18:58,880 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.NullPointerException: null
WARN [SharedPool-Worker-1] 2017-01-09 05:19:14,220 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.NullPointerException: null
WARN [SharedPool-Worker-1] 2017-01-09 05:19:29,876 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
WARN [SharedPool-Worker-1] 2017-01-09 05:19:45,217 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
WARN [SharedPool-Worker-2] 2017-01-09 05:20:00,241 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {}
问题3：关于泄漏的定期警告（这是在维修期间） - 我们在一周前发生了OOM事件：
INFO [Thread-9] 2017-01-09 15:01:11,828 RepairSession.java:237 - [repair #7593ad41-d67c-11e6-a76f-f1d6fd321f0d] new session: will sync ahldataslave1.bos.manhattan.aspect-cloud.net/10.184.8.151, /10.184.8.137, /10.184.8.219, /10.184.8.224 on range [(4815877001221178276,4825317823493984218], (3375739423664266689,3378070249246945856], (-7393599445401300009,-7393031105803218575], (-8130357422229977058,-8127934975152946543], (-4611683212167667055,-4597307445147354114], (4829191785668845783,4847928037092763493], (5522544388079333104,5528633388915935404], (1021118078158195547,1032730403399741255], (-20739833830226207,-14719054523073734], (-6205449048248449637,-6203730539323274527], (45435932670861622,69554714149031706], (148647262653106287,181904829526383862], (-854190757691833472,-854087983203765538], (-7714687510919982336,-7700463340346726648], (-360739232930366185,-339249720009847821], (524595904743716879,537371267474489752], (8185882494697143291,8187345181917158585], (181904829526383862,190096166859570216], (-1152328644109499972,-1134711681717758961], (-586016578220869954,-566687033455187734], (-4597307445147354114,-4541034774723552824], (-4488655272471923458,-4481233629800439836], (3111053603069111732,3121096557789915147], (-1167831376731724655,-1152328644109499972], (-6302582962631272172,-6290099341099600242], (8082429902489189721,8084233962135014968], (-2781993705200227829,-2777492754784519178], (5996816756578981045,6022011570976925279], (4404919629779433734,4422232637329318322], (-643317786131863042,-611582394998065454], (4514099138058135901,4514906323961126580], (7273471227296986256,7316794306200362675], (4462984376163219141,4464984714163480487], (-2390509810947844616,-2382282465250439153], (-7773729052257068390,-7772926086336061342], (-161569209320145017,-149914302996366392], (1392011488470148469,1403436343776848669], (546940383302067460,574197777751440495], (6624568645457217871,6629411218351717560], (-1134711681717758961,-1133755953039077472], (-7248281701751445296,-7243217407431962906], (2374653705052569734,2377396978942737768], (5619550734131192401,5628464353424177974], (-192925818286153751,-181902834109073167], (5896301416900157412,5911734690958039138], (4422232637329318322,4462984376163219141], (-4524564105581793254,-4504900851715014673], (-2115528876758613050,-2112322288682500569], (5504098260114685179,5515264579319185925], (35614519826275404,45435932670861622], (-855913784497695841,-854190757691833472], (5736245318761313405,5738445441111493493], (4807463771329802801,4815877001221178276], (-2884278504117308670,-2881329894351532736], (-207545241327995479,-207389332352346254], (-181902834109073167,-172819893538312319], (-7636862515759014032,-7626686102011136779], (-8140870857704067579,-8130357422229977058], (-2342122616740401439,-2335433185444036860], (-2881329894351532736,-2880349360092682175], (-611582394998065454,-587918354120388686], (-7320730625612121443,-7317644215584737598], (-13045270760172222,8260384715982984], (-8668935926714005039,-8657580917928283146], (8260384715982984,35614519826275404], (2861067483327829799,2864748323206840859], (3127114336475141934,3139163273099443478], (-7657420721193956821,-7636862515759014032], (1032730403399741255,1032951438204062821], (-1133755953039077472,-1130095656542041882], (5515264579319185925,5522544388079333104], (6826479291574328528,6828593995008227175], (658600138550738740,680809612853177633], (2364271158745468459,2374653705052569734], (2864748323206840859,2868871349367827459], (-7741910906325473118,-7738712583959975589], (2749562470526527976,2765502980561577063], (4569656624441412194,4581277061087905130], (-1130095656542041882,-1102822102812415509], (2357686856375407577,2364271158745468459], (2875146491882456097,2913694653712675567], (-2365721528619247416,-2362417204623405805], (-7700463340346726648,-7657420721193956821], (-1545400958003324469,-1527145992709891047], (6595468060656389384,6611122695943865425], (-2362417204623405805,-2360989147149959339], (680809612853177633,691564499046700936], (1048818468740031469,1051848236340749987], (-6290099341099600242,-6284139189426371408], (-430511209136180745,-417631748808875679], (-2127191227331961211,-2115528876758613050], (-587918354120388686,-586016578220869954], (5950659959556897324,5972536818860340742], (7228681112293176899,7273471227296986256], (-92115275177849392,-73078256610822906], (-2150182737057335813,-2131084968707606517], (-5931891237188071234,-5925084712702354100], (1854063092500381970,1894128417137542926], (3139163273099443478,3144353879892041737], (2795130016444690338,2849989872521310320], (69554714149031706,73259277905023853], (-198884270246085916,-192925818286153751], (2301867852127537258,2344138511230415627], (-2360989147149959339,-2342122616740401439], (2868871349367827459,2875146491882456097], (-8520120040026037727,-8484758828657908805], (1044175704807438060,1048818468740031469], (3658700192210846721,3709106004630729567], (-8715352510373593584,-8695484052773103051], (4825317823493984218,4829191785668845783], (5881177751245354064,5896301416900157412], (-1557733846751863547,-1545400958003324469], (3764340229927726801,3772352737530248438], (5723087139361841478,5736245318761313405], (-339249720009847821,-325063001525001753], (4581277061087905130,4585591969590606261], (2849989872521310320,2861067483327829799], (-201641945389990466,-198884270246085916], (3062119490734517386,3070339219117072282], (6611989977644669113,6624568645457217871], (-6303561532875621980,-6302582962631272172], (-2880349360092682175,-2815907193866317595], (-2210673969380244356,-2200655423201071981], (6920406973111913689,6944396390330370813], (3070339219117072282,3089918739923672617], (1018479337556218583,1021118078158195547], (-22373917471115570,-20802161777184740], (2349224556257120691,2357686856375407577], (1610291395978708348,1610350862682840167], (-7774308296105089774,-7773729052257068390], (-207389332352346254,-201641945389990466], (-4504900851715014673,-4488655272471923458], (-913362829471550684,-900550299179500948], (574197777751440495,575687259162456966], (-209852186312631406,-207545241327995479], (3961560378473489131,4029122901768525705], (1059898575782522398,1065913666232086022], (6611122695943865425,6611989977644669113], (-2815907193866317595,-2781993705200227829], (-8149621014270042242,-8140870857704067579], (-999997390934570392,-998678768267637866], (2377396978942737768,2384267095612056863], (-2267001694418802171,-2210871230482048364], (-2131084968707606517,-2127191227331961211], (3144353879892041737,3183343610940682057], (-2210871230482048364,-2210673969380244356], (4399306102077085676,4404919629779433734], (537371267474489752,546940383302067460], (-7722143244670988745,-7714687510919982336], (3754196018068729018,3763367677110000435], (-5925084712702354100,-5919412889821651249], (4585591969590606261,4590124192625756967], (-1189315862160445133,-1167831376731724655], (-14719054523073734,-13045270760172222], (3772352737530248438,3791853264732937441], (-149914302996366392,-138596249554837825], (-1459586834232833970,-1455782887662096513], (-7626686102011136779,-7613607518487725198], (-2370342707880863831,-2365721528619247416], (1051848236340749987,1059898575782522398], (575687259162456966,587747968172054843], (1846675642540253784,1854063092500381970], (5911734690958039138,5950659959556897324], (3709106004630729567,3754196018068729018], (2274687735800823626,2301867852127537258], (3763367677110000435,3764340229927726801], (2765502980561577063,2768302382460911776], (-4835698441475193016,-4821740368643417588], (-4840592682190099042,-4835698441475193016], (-900550299179500948,-898509802837897450], (5972536818860340742,5996816756578981045], (7316794306200362675,7316948326538779302], (-20802161777184740,-20739833830226207], (-8695484052773103051,-8693825425156218047], (-6228811049824135380,-6205449048248449637], (-8522744988679262888,-8520120040026037727], (-1527145992709891047,-1522580222898594256], (1065913666232086022,1070185844169898505], (3121096557789915147,3127114336475141934]] for manhattan.[config, messages, rcmessages, csseedmessages, metadata, rqmessages]
ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@8520181) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@110489339:/data/manhattan/messages-5e11ec20d49411e6ac36cb64da3c5bcf/ma-1-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2aa6af1) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1202250944:/data/manhattan/rcmessages-5f65e360d49411e6ac36cb64da3c5bcf/ma-1-big was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2017-01-09 15:01:12,481 Ref.java:202 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5feebbbd) to class org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1990550669:/data/manhattan/rcmessages-5f65e360d49411e6ac36cb64da3c5bcf/ma-2-big was not released before the reference was garbage collected
问题4：修复失败（无法完成修复） - 验证失败。
INFO [Thread-9] 2017-01-09 15:01:12,496 RepairSession.java:237 - [repair #75f99b00-d67c-11e6-a76f-f1d6fd321f0d] new session: will sync ahldataslave1.bos.manhattan.aspect-cloud.net/10.184.8.151, /10.184.8.136, /10.184.8.219, /10.184.8.206 on range [(7070473513877450523,7071785512108503562], (7540323476844137856,7551069853951512536], (6496851087781126402,6506947825241172690], (4956723654110177663,4997988272614120716], (6159728418399742238,6161785046042182181], (-6111201816165886930,-6109358652076682027], (6765434292326999735,6786216263996459971], (5670991529662923445,5678922736379606122], (-8368452819419163190,-8352559832808839636], (-3579114434972148656,-3552744642655983590], (7574005965138357735,7575976736757457093], (3443342880213529041,3448513378658973391], (-6374441519731158925,-6368083975098827965], (1506273267531550913,1508155162580364084], (-3841814855559153943,-3835938678769454001], (9170965416227969215,9181174655986386320], (-6589650497263617394,-6578994961053045565], (-7526495575912144496,-7518448902193774650], (3448513378658973391,3466857105667177973], (7553098986919382391,7574005965138357735], (-7927877550785715833,-7927648953253692861], (8525761079018902513,8533541215797136240], (-9020129600263662082,-9010501329825003246], (1090724998362773160,1097939939665235297], (6495099300248039972,6496851087781126402], (6161785046042182181,6230901034893815059], (-7463338857564269606,-7459175745770455573], (-5365113529531278290,-5358403398116793300], (2249898065901245108,2256752991177466019], (427458615961451939,456792456721828910], (-7518448902193774650,-7476577165771266744], (8496964666711176665,8503454084084785332], (1508155162580364084,1531943799366406071], (1623939448991457330,1628591069075100791], (2500969373198505595,2510256322219581447], (1487072216030226303,1506273267531550913], (-8842784826526224555,-8840216173643212475], (2233348668129647030,2247658708769712814], (-8289206614243730701,-8287074180401539509], (-7476577165771266744,-7470389682785988771], (3509709684061378867,3512938839959595162], (1296769759309759534,1302830049359124705], (7575976736757457093,7612135532866611803], (7539950122318008545,7540323476844137856], (-1860975172751451123,-1853676643512030698], (-8116786947316919605,-8107369675963369796], (-5576866678991811211,-5548824138592035245], (-9211881728677655622,-9209566963086044390], (7489367377277174970,7511259639253940249], (1302830049359124705,1310010318365119318], (-8750733220711433842,-8741408722769214425], (-6060376367842620341,-6049131555099770258], (2476529278549364684,2476659930520260878], (-4418742343319617797,-4384333951548488071], (1930660747072472501,1946687557062422024], (-3702380150896445622,-3691910222307497728], (-4419703203609450987,-4418742343319617797], (7551697213870628887,7553098986919382391], (-7909869864636951454,-7904683595914140744], (2492882420790179151,2500969373198505595], (759685512451005610,773693246847159219], (722689136592207691,730821328907957274], (5219025897237634110,5229566135848567231], (4149517035748854868,4162833917677490963], (1326132699179277905,1349642653396670633], (-3552744642655983590,-3549073980410969181], (-4456698363145934960,-4419703203609450987], (-5482989026069906323,-5481213659503324794], (279535406932511121,297415597645494493], (-9204266745720873498,-9161795161892156657], (6786216263996459971,6786883290959946615], (1635462812102810232,1638975594001298412], (1531943799366406071,1546279755200334429], (-3851094911406737954,-3841814855559153943], (8488130353439482734,8496964666711176665], (9146936609495586767,9148689706364420814], (-3674452659583777301,-3669778494255064100], (1546279755200334429,1554992050513055527], (2969501640373218313,2975603144417796179], (-3865994019611705472,-3851094911406737954], (1119818899797211952,1121114266048498658], (5555981883036126656,5559877922798642583], (-9161795161892156657,-9148239227447645822], (-1988480938545038453,-1972594091279542463], (-8457443005098123276,-8426466451426413266], (7527886006442742638,7539950122318008545], (1482724986864293552,1487072216030226303], (218814708185728764,222373457441562248], (-1844742172330357357,-1834186452017037646], (-7914786200548736052,-7909869864636951454], (1554992050513055527,1556471756149638584], (2247658708769712814,2249241162600622119], (2249241162600622119,2249898065901245108], (-6368083975098827965,-6357129727004451888], (1349642653396670633,1372390693080139703], (-5628093303198956105,-5602276528881256607], (-5602276528881256607,-5576866678991811211], (-1972594091279542463,-1932380490869749103], (6746187069375198686,6759453964579921534], (-9209566963086044390,-9204266745720873498], (2969169700471726709,2969501640373218313], (-1846725725884736711,-1844742172330357357], (8466845312658849111,8478632986527402947], (6786883290959946615,6786909234664957611], (-8105207391882034800,-8058995460560987960], (-5481213659503324794,-5480151738028046165], (6152943806705127166,6159728418399742238], (-7931416471770778308,-7927877550785715833], (-7444305315017293180,-7438698090368547807], (-5405765462772655927,-5365113529531278290], (7468218535666030613,7489367377277174970], (7370990959551251700,7384554095789173771], (9181174655986386320,9197356813903415489], (-7891619558235712999,-7886926665775314135], (4863660993721847898,4876110806493712229], (5546813895398001993,5555981883036126656], (6793672076817085638,6817463007248949484], (730821328907957274,741073317079317196], (1285297621466628116,1296769759309759534], (-6109358652076682027,-6097749005908966679], (-3160359268440132710,-3150255443388600113], (-6067028212437311851,-6060376367842620341], (3512938839959595162,3531669213417260709], (-1641724835017736571,-1630212592242213627], (-5631109618795517413,-5628093303198956105], (6759453964579921534,6765434292326999735], (8027648385044762960,8030142266857918084], (1264145341476479882,1285297621466628116], (1116504555019555851,1119818899797211952], (-3139664283152392978,-3138393885745606369], (-1652241560457928053,-1641724835017736571], (8503454084084785332,8525761079018902513], (-1853676643512030698,-1846725725884736711], (1628591069075100791,1635462812102810232], (-8840216173643212475,-8817712049837887184], (-6357129727004451888,-6355060010614218705], (-6097749005908966679,-6088616433680186631], (-7904683595914140744,-7891619558235712999], (-4952793941753266051,-4937049631932742604], (-3150255443388600113,-3139664283152392978], (2512801900471921714,2516325796899709313], (2510256322219581447,2512801900471921714], (-5638226176228799616,-5631109618795517413], (-7886926665775314135,-7882937045572193145], (6786909234664957611,6793672076817085638], (-3691910222307497728,-3674452659583777301], (-4867940821711541762,-4866946150328928181], (7071785512108503562,7081793743588959944], (4876110806493712229,4879643371134562502], (6138845154266790671,6152943806705127166], (-6355060010614218705,-6332373169802347251], (-6125214696594291714,-6111201816165886930], (-7470389682785988771,-7463338857564269606], (-3598204681868075295,-3579114434972148656], (-8107369675963369796,-8105207391882034800], (7612135532866611803,7641727005250566009], (9148689706364420814,9170965416227969215], (6506947825241172690,6515762919020447377], (-6088616433680186631,-6067028212437311851], (741073317079317196,759685512451005610], (-8352559832808839636,-8344010281020219217], (-6049131555099770258,-6043632332003790850], (3494099643169044463,3509709684061378867], (3466857105667177973,3494099643169044463], (7551069853951512536,7551697213870628887], (2476659930520260878,2492882420790179151], (-8842911646480810622,-8842784826526224555], (193808271419729511,198960416318160202], (-4873803694032119901,-4867940821711541762], (-7569775426219162591,-7569610743922997927], (4730391117872973381,4732570096640666262], (9137762516596245936,9146936609495586767], (-9148239227447645822,-9137147760354721369], (-8296203956052422690,-8289206614243730701], (198960416318160202,218814708185728764]] for manhattan.[config, messages, rcmessages, csseedmessages, metadata, rqmessages]
WARN [SharedPool-Worker-2] 2017-01-09 15:01:38,219 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.NullPointerException: null
WARN [SharedPool-Worker-2] 2017-01-09 15:04:41,678 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2461) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_111]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]