每个节点向现有数据集群集添加新的单个令牌并且数据传输无效。随后的过程如下所述。如果我所遵循的流程是错误的,请更新我。谢谢
我们的AWS EC2 Datacenter中有3个单令牌范围数据传输节点,启用了搜索和图形。我们计划在数据中心添加3个节点。我们目前正在为密钥空间使用DseSimpleSnitch和Simple网络拓扑。我们当前的复制因子也是2 。
节点1:10.10.1.36
节点2:10.10.1.46
节点3:10.10.1.56
cat /etc/default/dse | grep -E 'GRAPH_ENABLED=|SOLR_ENABLED='
GRAPH_ENABLED=1
SOLR_ENABLED=1
数据中心:SearchGraph
Address Rack Status State Load Owns Token
10.10.1.46 rack1 Up Normal 760.14 MiB ? -9223372036854775808
10.10.1.36 rack1 Up Normal 737.69 MiB ? -3074457345618258603
10.10.1.56 rack1 Up Normal 752.25 MiB ? 3074457345618258602
步骤(1)为了首先将3个新节点添加到我们的数据中心,我们更改了密钥空间拓扑,并提醒了网络。
1)改变了小报。 cat /etc/dse/cassandra/cassandra.yaml | grep endpoint_snitch: endpoint_snitch:GossipingPropertyFileSnitch
cat /etc/dse/cassandra/cassandra-rackdc.properties |grep -E 'dc=|rack='
dc=SearchGraph
rack=rack1
2) (a)关闭所有节点,然后重新启动它们。
(b)在每个节点上运行顺序修复和nodetool清理。
3)更改了密钥空间拓扑。
ALTER KEYSPACE tech_app1 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'SearchGraph' : 2};
ALTER KEYSPACE tech_app2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'SearchGraph' : 2};
ALTER KEYSPACE tech_chat WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'SearchGraph' : 2};
参考:http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsChangeKSStrategy.html,http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html
步骤(2)为了更新令牌范围和设置新的cassandra节点,我们按照以下流程进行操作。
1)重新计算令牌范围
root@ip-10-10-1-36:~# token-generator
DC#1:
Node #1: -9223372036854775808
Node #2: -6148914691236517206
Node #3: -3074457345618258604
Node #4: -2
Node #5: 3074457345618258600
Node #6: 6148914691236517202
2)在新节点上安装了Datastax企业相同的版本。
3)停止节点服务并清除数据。
4)(a)按以下方式为新节点分配令牌范围。
Node 4: 10.10.2.96 Range: -2
Node 5: 10.10.2.97 Range: 3074457345618258600
Node 6: 10.10.2.86 Range: 6148914691236517202
4)(b)在每个新节点上配置cassandra.yaml:
节点4:
cluster_name: 'SearchGraph'
num_tokens: 1
initial_token: -2
parameters:
- seeds: "10.10.1.46, 10.10.1.56"
listen_address: 10.10.2.96
rpc_address: 10.10.2.96
endpoint_snitch: GossipingPropertyFileSnitch
节点5:
cluster_name: 'SearchGraph'
num_tokens: 1
initial_token: 3074457345618258600
parameters:
- seeds: "10.10.1.46, 10.10.1.56"
listen_address: 10.10.2.97
rpc_address: 10.10.2.97
endpoint_snitch: GossipingPropertyFileSnitch
节点6:
cluster_name: 'SearchGraph'
num_tokens: 1
initial_token: 6148914691236517202
parameters:
- seeds: "10.10.1.46, 10.10.1.56"
listen_address: 10.10.2.86
rpc_address: 10.10.2.86
endpoint_snitch: GossipingPropertyFileSnitch
5)改变了小报。
cat /etc/dse/cassandra/cassandra.yaml | grep endpoint_snitch:
endpoint_snitch: GossipingPropertyFileSnitch
cat /etc/dse/cassandra/cassandra-rackdc.properties |grep -E 'dc=|rack='
dc=SearchGraph
rack=rack1
6)每隔两分钟在每个新节点上启动DataStax Enterprise,并关闭consistent.rangemovement:
JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false
7)在新节点完全自举之后,使用nodetool移动以根据在步骤4(a)完成的令牌重新计算为现有节点分配新的initial_token。每个节点一次完成一个过程。
On Node 1(10.10.1.36) : nodetool move -3074457345618258603
On Node 2(10.10.1.46) : nodetool move -9223372036854775808
On Node 3(10.10.1.56) : nodetool move 3074457345618258602
数据中心:SearchGraph
Address Rack Status State Load Owns Token
10.10.1.46 rack1 Up Normal 852.93 MiB ? -9223372036854775808
10.10.1.36 rack1 Up Moving 900.12 MiB ? -3074457345618258603
10.10.2.96 rack1 UP Normal 465.02 KiB ? -2
10.10.2.97 rack1 Up Normal 109.16 MiB ? 3074457345618258600
10.10.1.56 rack1 Up Moving 594.49 MiB ? 3074457345618258602
10.10.2.86 rack1 Up Normal 663.94 MiB ? 6148914691236517202
发布更新:
但我们在加入节点时遇到以下错误。
AbstractSolrSecondaryIndex.java:1884 - Cannot find core chat.chat_history
AbstractSolrSecondaryIndex.java:1884 - Cannot find core chat.history
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.business_units
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.feeds
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.feeds_2
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.knowledegmodule
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.userdetails
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.userdetails_2
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.vault_details
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.workgroup
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.feeds
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.knowledgemodule
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.organizations
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.userdetails
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.vaults
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.workgroup
节点加入失败,出现以下错误:
ERROR [main] 2017-08-10 04:22:08,449 DseDaemon.java:488 - Unable to start DSE server.
com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.plugin.SolrContainerPlugin
Caused by: java.lang.IllegalStateException: Cannot find secondary index for core ekamsearch.userdetails_2, did you create it?
If yes, please consider increasing the value of the dse.yaml option load_max_time_per_core, current value in minutes is: 10
ERROR [main] 2017-08-10 04:22:08,450 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin
之前是否有人遇到过这些错误或警告?
答案 0 :(得分:0)
令牌分配问题::
1) I had wrongly assigned token range in Step 4) (a). Assign token which
bisect or trisect the value which are generated using
"token-generator"
Node 4: 10.10.2.96 Range: -6148914691236517206
Node 5: 10.10.2.97 Range: -2
Node 6: 10.10.2.86 Range: 6148914691236517202
Note : We don't need to change the token range of existing nodes in data
center.No need to follow procedure in Step 7 which i have mentioned
above.
Solr问题已解决:无法找到cor ::
Increased load_max_time_per_core value in dse.yaml configuration file,
still i was receving the error.Finalys solved the issue
by following method
1) Started the new nodes as non-solr and wait for all cassandra data
to migrate to joining nodes.
2) Add the parameter auto_bootstrap: False directive to the
cassandra.yaml file
3) Re-start the same nodes after enabling solr. Changed parameter
SOLR_ENABLED=1 in /etc/default/dse
3) Re-index in all new joined nodes. I had to reloaded all core
required with the reindex=true and distributed=false parameters in
new joined nodes.
Ref : http://docs.datastax.com/en/archived/datastax_enterprise/4.0/datastax_enterprise/srch/srchReldCore.html