There are two common types of computer clusters:
1) Load balancing computer clusters (loadbalancers, multiple web and database servers)
2) High-availability computer clusters (eg..Redhat cluster stack- corosync,heartbeat,pacemaker etc)
Category 2 ) High-availability computer clusters strive to minimize the downtime of the service
provided and not so much to improve the overall performance of the service.
active/active
active/passive
N + 1 =The N over 1 configuration
implies the presence of N cluster members in an active/active configuration with one cluster member in backup or hot standby. The standby cluster member is ready to take over any of the failed cluster node responsibilities at
any given time.
The cluster software stack must provide two significant features:
1)cluster communication : uses tcp or udp broadcast/multicast
2)cluster resource management: Cluster resource management starts, stops, and manages applications in the cluster in general. It relies on the cluster communication layer to get the information it requires about cluster members and their status, and takes cluster resource
actions based on that
Cluster resource manager components: -
TOTEM provides cluster group communication. TOTEM provides
cluster messaging and membership services. The cluster messaging service is the communication process among cluster members. This communication process is called a
heartbeat and the formation cluster members adopt is called a ring.
TOTEM uses the broadcast method to send messages
to all recipients simultaneously.
TOTEM protocol introduces a special token, passed among the cluster members. Only the cluster member that possesses this TOKEN can broadcast messages to other cluster members. Virtual Synchrony concept was extended to allow continued operation with a so-called temporary transitional cluster configuration with reduced membership in case of cluster member failures
The Membership Protocol detects cluster member failures and rebuilds the cluster configuration, leaving out failed cluster members. A new passing token is also generated
Another very important process managed by the cluster communication layer is
called Quorum. Quorum is the minimum number of cluster member votes required to perform a cluster operation Some cluster communication software solutions support
an additional quorum disk that counts as an additional so-called tie-breaker vote.
The quorum disk is very useful and also required in an even-node cluster configuration. In a two-node cluster, only two votes are available from the cluster nodes, which means that, in the event of a cluster failure, the operational cluster node does not have quorum majority and
can't make decisions. The quorum disk is a shared storage disk to which all the cluster
members have access. All the cluster members use the quorum disk for heartbeat
communication and store cluster configuration data in it. Based on the activity on the quorum disk, cluster members can run their cluster communication via the quorum
disk, and the quorum disk can also count as a vote in the quorum process. The quorum
disk is a must in an even-node cluster configuration.
SPLIT Brain is a condition in the cluster where the cluster is split into two sides, each side thinking
that the other side is dead or inactive. In this situation, each side proceeds to take
over the resources as if the other side no longer exists. To prevent such scenarios,
an additional mechanism called FENCING is used. Before any of the split sides of the
cluster takes over the resources, the fencing mechanism fences the other side by
issuing a configured action of reboot or power-off on the cluster nodes, thus making
sure that the cluster nodes on the other side are really dead. Once successful fencing
is confirmed, the operational side can safely start the resources without any data
corruption or disruption. types of fencing: resource-level fencing and node-level fencing. Resource-level fencing is the way the cluster disables a specific or problematic node to access a specific resource. This is usually used on shared storage resources, where a
problematic node is denied access to certain shared storage. More commonly used
is node-level fencing. Node-level fencing is a way to make sure a node does not run
any resources at all.
CentOS 6 and CentOS 7. The only difference between the two versions is the cluster resource
manager software used. This means that both solutions will use the same cluster communication software, called Corosync. Corosync provides cluster communication features and
capabilities. Corosync provides TOTEM with virtual synchrony, Quorum.
Since cluster resource manager software must run on top of Corosync, this is where
the road splits into two. There are two cluster resource manager software options
to use—either RGManager or Pacemaker . Generally, Pacemaker has more features
than RGManager. RGManager also requires the CMAN software as the middle layer
between it and Corosync. CMAN is the communication layer between Corosync and
RGManager, also providing some additional functionality. It is an interesting fact
that the official Red Hat Cluster software suite in RHEL 6 version includes Corosync,
CMAN, and RGManager; in RHEL 7, however, this was changed to Corosync and
Pacemaker.https://alteeve.ca/w/Clustering
Configuration
how to calculate quorum?, the formula is:
quorum=peep(expected votes/2 + 1)
peep () function means allways round down
So if for example you hace a cluster formed by 3 nodes:
quorum=peep( 3 / 2 + 1)
quorum=peep( 1'5 + 1)
quorum=peep(2'5)
quorum=2
7 nodes..
quorum=peep (7 /2 +1 )
quorum=peep ( 3'5 +1 )
quorum=4
With cman_tool we can change the number of expected votes, lets say that we have and emergency and need to get a cluster quorate we can change the expected votes.
For example in a 3 node cluster we need to get a cluster working with only 1 node:
[root@centos ~]# clustat
Cluster Status for newcluster @ Thu Jul 18 17:21:33 2013
Member Status: Quorate
[root@centos ~]# cman_tool status
Version: 6.2.0
Config Version: 5
Cluster Name: newcluster
Cluster Id: 43508
Cluster Member: Yes
Cluster Generation: 60
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 8
Flags:
Ports Bound: 0 177
Node name: centosclu
Node ID: 1
Multicast addresses: 239.192.169.158
Node addresses: 1.1.1.2
[root@centos ~]# ccs_tool lsnode
Cluster name: newcluster, config_version: 6
Nodename Votes Nodeid Fencetype
centosclu 1 1
centos2clu 1 2
centos3clu 1 3
So lets say we loose centos2clu and centos3clu, and we need to start the cluster only with centosclu node:
we stop 1 node
[root@centos ~]# cman_tool status
Nodes: 2
Expected votes: 3
Total votes: 2
Node votes: 1
Quorum: 2
we stop 2 nodes:
[root@centos ~]#
Message from syslogd@centos at Jul 18 17:25:19 ...
rgmanager[2847]: #1: Quorum Dissolved
[root@centos ~]# cman_tool status
Membership state: Cluster-Member
Nodes: 1
Expected votes: 3
Total votes: 1
Node votes: 1
Quorum: 2 Activity blocked
[root@centos ~]# clustat
Service states unavailable: Operation requires quorum
Cluster Status for newcluster @ Thu Jul 18 17:26:22 2013
Member Status: Inquorate
So if we wan't to force the cluster to work with only one node we can change the expected votes:
[root@centos ~]# cman_tool expected -e 1
[root@centos ~]# clustat
Cluster Status for newcluster @ Thu Jul 18 17:27:37 2013
Member Status: Quorate
[ root@centos ~]# cman_tool status
Nodes: 1
Expected votes: 1
T otal votes: 1
Node votes: 1
Quorum: 1
Once we recover the nodes we need to leave the expected votes back to normal otherwise we can have serious problems with data integrity.
If we wan't to modify how many votes a node adds to a cluster we have several ways, for example manually editing the cluster.conf file:
root@centos cluster]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
We can get info on all the parameters of the cluster.conf file from this html file:
/usr/share/doc/cman-3.0.12.1/cluster_conf.html
Once we modify the file we need to increment by one the config_version="7" number, and then distribute and activate the file using for example cman_tool:
[root@centos cluster]# cman_tool version -r
You have not authenticated to the ricci daemon on centos3clu
Password:
You have not authenticated to the ricci daemon on centos2clu
Password:
You have not authenticated to the ricci daemon on centosclu
Password:
[root@centos cluster]# ccs_tool lsnode
Cluster name: newcluster, config_version: 7
Nodename Votes Nodeid Fencetype
centosclu 2 1
centos2clu 1 2
centos3clu 1 3
[root@centos cluster]# cman_tool status
Version: 6.2.0
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 4
Node votes: 2
Quorum: 3
we can also modify expected votes to 4, so it all looks how it does:
[root@centos cluster]# cman_tool expected -e 4
[root@centos cluster]# cman_tool status
Nodes: 3
Expected votes: 4
Total votes: 4
Node votes: 2
Quorum: 3
Active subsystems: 8
Flags:
Ports Bound: 0 177
Node name: centosclu
Node ID: 1
Multicast addresses: 239.192.169.158
Node addresses: 1.1.1.2
Another exercise, we are going to give centos node 3 votes so even if we loose centos2 and centos3, we still have qourom tu run with centos.
We modify the file /etc/cluster/cluster.conf like before, and distribute it to all nodes, using ccs this time:
[root@centos4 ~]# ccs -h centos --sync --activate
We can check we have the new version working with 3 votes for node1:
[root@centos3 cluster]# cman_tool status
Version: 6.2.0
Config Version: 10
Cluster Name: newcluster
Cluster Id: 43508
Cluster Member: Yes
Cluster Generation: 192
Membership state: Cluster-Member
Nodes: 3
Expected votes: 4
Total votes: 5
Node votes: 1
Quorum: 3
No we take down centos2 and centos3
[liquid@liquid-ibm:~]$ virsh destroy centos2 (07-18 11:27)
Domain centos2 destroyed
[liquid@liquid-ibm:~]$ virsh destroy centos3 (07-18 18:07)
Domain centos3 destroyed
On centos logs we can see:
Jul 18 18:08:00 centos rgmanager[2267]: State change: centos2clu DOWN
Jul 18 18:08:00 centos fenced[1866]: fencing node centos3clu
Jul 18 18:08:00 centos rgmanager[2267]: State change: centos3clu DOWN
[root@centos ~]# clustat
Cluster Status for newcluster @ Thu Jul 18 18:10:03 2013
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
centosclu 1 Online, Local
centos2clu 2 Offline
centos3clu 3 Offline
[root@centos ~]# cman_tool status
Version: 6.2.0
Config Version: 10
Cluster Name: newcluster
Cluster Id: 43508
Cluster Member: Yes
Cluster Generation: 196
Membership state: Cluster-Member
Nodes: 1
Expected votes: 4
Total votes: 3
Node votes: 3
Quorum: 3
- See more at: http://www.hpuxtips.es/?q=content/part5quorum-my-study-notes-red-hat-certificate-expertise-clustering-and-storage-management#sthash.sacx9AAD.dpuf
how to calculate quorum?, the formula is:
quorum=peep(expected votes/2 + 1)
peep () function means allways round down
So if for example you hace a cluster formed by 3 nodes:
quorum=peep( 3 / 2 + 1)
quorum=peep( 1'5 + 1)
quorum=peep(2'5)
quorum=2
7 nodes..
quorum=peep (7 /2 +1 )
quorum=peep ( 3'5 +1 )
quorum=4
With cman_tool we can change the number of expected votes, lets say that we have and emergency and need to get a cluster quorate we can change the expected votes.
For example in a 3 node cluster we need to get a cluster working with only 1 node:
[root@centos ~]# clustat
Cluster Status for newcluster @ Thu Jul 18 17:21:33 2013
Member Status: Quorate
[root@centos ~]# cman_tool status
Version: 6.2.0
Config Version: 5
Cluster Name: newcluster
Cluster Id: 43508
Cluster Member: Yes
Cluster Generation: 60
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 8
Flags:
Ports Bound: 0 177
Node name: centosclu
Node ID: 1
Multicast addresses: 239.192.169.158
Node addresses: 1.1.1.2
[root@centos ~]# ccs_tool lsnode
Cluster name: newcluster, config_version: 6
Nodename Votes Nodeid Fencetype
centosclu 1 1
centos2clu 1 2
centos3clu 1 3
So lets say we loose centos2clu and centos3clu, and we need to start the cluster only with centosclu node:
we stop 1 node
[root@centos ~]# cman_tool status
Nodes: 2
Expected votes: 3
Total votes: 2
Node votes: 1
Quorum: 2
we stop 2 nodes:
[root@centos ~]#
Message from syslogd@centos at Jul 18 17:25:19 ...
rgmanager[2847]: #1: Quorum Dissolved
[root@centos ~]# cman_tool status
Membership state: Cluster-Member
Nodes: 1
Expected votes: 3
Total votes: 1
Node votes: 1
Quorum: 2 Activity blocked
[root@centos ~]# clustat
Service states unavailable: Operation requires quorum
Cluster Status for newcluster @ Thu Jul 18 17:26:22 2013
Member Status: Inquorate
So if we wan't to force the cluster to work with only one node we can change the expected votes:
[root@centos ~]# cman_tool expected -e 1
[root@centos ~]# clustat
Cluster Status for newcluster @ Thu Jul 18 17:27:37 2013
Member Status: Quorate
[ root@centos ~]# cman_tool status
Nodes: 1
Expected votes: 1
T otal votes: 1
Node votes: 1
Quorum: 1
Once we recover the nodes we need to leave the expected votes back to normal otherwise we can have serious problems with data integrity.
If we wan't to modify how many votes a node adds to a cluster we have several ways, for example manually editing the cluster.conf file:
root@centos cluster]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
We can get info on all the parameters of the cluster.conf file from this html file:
/usr/share/doc/cman-3.0.12.1/cluster_conf.html
Once we modify the file we need to increment by one the config_version="7" number, and then distribute and activate the file using for example cman_tool:
[root@centos cluster]# cman_tool version -r
You have not authenticated to the ricci daemon on centos3clu
Password:
You have not authenticated to the ricci daemon on centos2clu
Password:
You have not authenticated to the ricci daemon on centosclu
Password:
[root@centos cluster]# ccs_tool lsnode
Cluster name: newcluster, config_version: 7
Nodename Votes Nodeid Fencetype
centosclu 2 1
centos2clu 1 2
centos3clu 1 3
[root@centos cluster]# cman_tool status
Version: 6.2.0
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 4
Node votes: 2
Quorum: 3
we can also modify expected votes to 4, so it all looks how it does:
[root@centos cluster]# cman_tool expected -e 4
[root@centos cluster]# cman_tool status
Nodes: 3
Expected votes: 4
Total votes: 4
Node votes: 2
Quorum: 3
Active subsystems: 8
Flags:
Ports Bound: 0 177
Node name: centosclu
Node ID: 1
Multicast addresses: 239.192.169.158
Node addresses: 1.1.1.2
Another exercise, we are going to give centos node 3 votes so even if we loose centos2 and centos3, we still have qourom tu run with centos.
We modify the file /etc/cluster/cluster.conf like before, and distribute it to all nodes, using ccs this time:
[root@centos4 ~]# ccs -h centos --sync --activate
We can check we have the new version working with 3 votes for node1:
[root@centos3 cluster]# cman_tool status
Version: 6.2.0
Config Version: 10
Cluster Name: newcluster
Cluster Id: 43508
Cluster Member: Yes
Cluster Generation: 192
Membership state: Cluster-Member
Nodes: 3
Expected votes: 4
Total votes: 5
Node votes: 1
Quorum: 3
No we take down centos2 and centos3
[liquid@liquid-ibm:~]$ virsh destroy centos2 (07-18 11:27)
Domain centos2 destroyed
[liquid@liquid-ibm:~]$ virsh destroy centos3 (07-18 18:07)
Domain centos3 destroyed
On centos logs we can see:
Jul 18 18:08:00 centos rgmanager[2267]: State change: centos2clu DOWN
Jul 18 18:08:00 centos fenced[1866]: fencing node centos3clu
Jul 18 18:08:00 centos rgmanager[2267]: State change: centos3clu DOWN
[root@centos ~]# clustat
Cluster Status for newcluster @ Thu Jul 18 18:10:03 2013
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
centosclu 1 Online, Local
centos2clu 2 Offline
centos3clu 3 Offline
[root@centos ~]# cman_tool status
Version: 6.2.0
Config Version: 10
Cluster Name: newcluster
Cluster Id: 43508
Cluster Member: Yes
Cluster Generation: 196
Membership state: Cluster-Member
Nodes: 1
Expected votes: 4
Total votes: 3
Node votes: 3
Quorum: 3
- See more at: http://www.hpuxtips.es/?q=content/part5quorum-my-study-notes-red-hat-certificate-expertise-clustering-and-storage-management#sthash.sacx9AAD.dpuf
No comments:
Post a Comment