Disable SELinux on all nodes
setenforce 0
All nodes are under oracle virtual box:-
number of cluster nodes : 3
setup static ip for all 3 nodes and proper hostname
example
[root@c1 cluster]# cat /etc/hosts
127.0.0.1 localhost
192.168.0.55 c1.example.com c1
192.168.0.66 c2.example.com c2
192.168.0.77 c3.example.com c3
Setup proper gateway( setup similar config in other 2 nodes)
[root@c1]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=c1.example.com
GATEWAY=192.168.0.235
[root@c1 ]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=08:00:27:2E:24:80
TYPE=Ethernet
UUID=02e21921-20a5-4490-bf4a-052ab8cb6c11
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=192.168.0.55
NETMASK=255.255.255.0
NTP configuration: ON ALL 3 nodes. You must install and start the NTP time synchronization
service to maintain consistent time throughout the cluster:
[root@node-1 ~]# yum install ntp –y
Installation time is synchronized with official CentOS NTP
servers. If you want to change this, edit the NTP configuration file at /etc/
ntp.conf.
You must start the NTP service and make sure it starts at boot. In the
following screenshot, you can see the commands used:
[root@c1 ~]# service ntpd start
[root@c1 ~]# chkconfig ntpd on
Setting up corosync
setup iptables to allow cluster communication traffic among cluster nodes. You must add an iptables rule to allow UDP traffic on 5404 and 5405, and another rule to allow multicast traffic communication :
iptables -I INPUT -m state --state NEW -p udp -m multiport --dports 5404,5405 -j ACCEPT
iptables -I INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
The default Corosync cluster communication ports are 5404 and
5405, and the UDP protocol is used. The ports are configured in the
/etc/corosync/corosync.conf configuration file. If you want
to change the default Corosync communication ports, make sure you
change iptables accordingly.
service iptables save
yum install corosync -y #do it on all nodes
#generate corosync encryption key on any one node and transfer it to rest of nodes
#corosync-keygen
it might take long time so to speed up
dd if=/dev/random of=test123.log
Make backup of original file and rename on any one node.
[root@c1 ~]# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
#vim corosync.conf
Remove all commented-out lines—lines starting with #—and change the
secauth parameter to on.
Change the bindnetaddr parameter to the current cluster node's IP address.
* Remove all commented-out lines—lines starting with #—and change the
secauth parameter to on.
* Change the bindnetaddr parameter to the current cluster node's IP address.
* Change the to_syslog parameter to no. We will use a separate log file, so there's no need to duplicate the information in syslog.
Now scp the corosync.conf to all rest of nodes
On all nodes
#service corosync start
#chkconfig corosync on
To check if members have joined
[root@c3 ~]# corosync-objctl | grep member.*ip
runtime.totem.pg.mrp.srp.members.3.ip=r(0) ip(192.168.0.77)
runtime.totem.pg.mrp.srp.members.1.ip=r(0) ip(192.168.0.55)
runtime.totem.pg.mrp.srp.members.2.ip=r(0) ip(192.168.0.66)
You can stop corosync for some time on all nodes
#service corosync stop
Install CMAN on all nodes
yum install cman -y
Create cluster.conf file on any one node and copy to rest of nodes
cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="3" name="hacluster">
<logging debug="off"/>
<clusternodes>
<clusternode name="c1" nodeid="1"/>
<clusternode name="c2" nodeid="2"/>
<clusternode name="c3" nodeid="3"/>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="simple" nofailback="1" ordered="0" restricted="0">
<failoverdomainnode name="c1"/>
<failoverdomainnode name="c2"/>
<failoverdomainnode name="c3"/>
</failoverdomain>
</failoverdomains>
</rm>
</cluster>
The config_version parameter is the version of the CMAN
configuration file(it should be same on all nodes and should be increased by one+ when changed similar to serial number in bind zone configuration), and the name parameter is your preferred cluster name.
The debug parameter is set to off and should be turned on for debugging purposes.The clusternodes section is where the cluster nodes are configured. The clusternode parameter is defined by a name parameter, which is the FQDN or short node name, and the nodeid parameter, which is the ID of the cluster node.
Once the CMAN service has been successfully started on all cluster nodes, check the CMAN node status using the cman_tool nodes command. All three cluster nodes should be listed with status as M. In the following screenshot, you can see the output of the cman_tool nodes command issued on the node-1 cluster node:
[root@c1 ~]# cman_tool nodes
Node Sts Inc Joined Name
1 M 364 2015-08-26 10:53:04 c1
2 M 372 2015-08-26 10:54:14 c2
3 M 368 2015-08-26 10:53:18 c3
Let's take a look at each column in more detail:
• Node: This column shows the node ID number.
• Sts: This column shows the cluster node status, with M indicating that the
node is a joined cluster member and X indicating that the node is dead.
• Inc: This column is for debugging purposes only. It is the incarnation number.
• Joined: This column shows the time the cluster member joined the cluster.
• Name: This column shows the cluster node name as defined in the cluster.conf file.
Check quorum status:-
[root@c1 ~]# cman_tool status
Version: 6.2.0
Config Version: 3
Cluster Name: hacluster
Cluster Id: 10117
Cluster Member: Yes
Cluster Generation: 372
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 7
Flags:
Ports Bound: 0
Node name: c1
Node ID: 1
Multicast addresses: 239.192.39.172
Node addresses: 192.168.0.55
Nodes: This line is where the number of your cluster nodes
should be listed
Expected votes: This line shows the number of expected votes
in the currently active configuration
Total votes: This line shows the number of total votes provided
by the CMAN configuration file
Node votes: This line shows the number of votes the current
node has
Quorum: This line shows the number of votes required to reach
Quorum
The finishing touch for the CMAN installation and configuration process is the start of a service called ricci. "The ricci service provides on-the-fly CMAN configuration file distribution."
setup password for ricci user and start ricci on all nodes-
passwd ricci
Password: ********
service ricci start
chkconfig ricci on
Setup RG Manager on all nodes:-
iptables -I INPUT -m state --state NEW -p tcp -m multiport --dports 11111,21064,41966,41967,41968,41969 -j ACCEPT
service iptables save
service iptables restart
yum install rgmanager -y
service rgmanager start
chkconfig rgmanager on
Check cluster status:-
#clustat
#Member status: This line provides information about the
Quorum status.
#Member Name: This column shows the cluster node name as
defined in the cluster.conf configuration file.
#ID: This column shows the node ID number.
#Status: This column provides information about the node's
status. The output of the clustat command must list all
the cluster nodes as online, confirming that installation and
configuration of the cluster stack software on CentOS 6 were
successful and that you did a great job at it.
Edit cluster.conf file on any one node(you may need to manually create/copypaste same file on node which is offline) , make sure to increment configversion="" after making changes:-
[root@c1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="3" name="hacluster">
<logging debug="off"/>
<clusternodes>
<clusternode name="c1" nodeid="1"/>
<clusternode name="c2" nodeid="2"/>
<clusternode name="c3" nodeid="3"/>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="simple" nofailback="1" ordered="0" restricted="0">
<failoverdomainnode name="c1"/>
<failoverdomainnode name="c2"/>
<failoverdomainnode name="c3"/>
</failoverdomain>
</failoverdomains>
</rm>
</cluster>
To validate
[root@c1 ~]# ccs_config_validate
Configuration validates
Once it is found valid, you can propagate to other nodes
# cman_tool version -r
Note that, if a specific cluster node is not online, the configuration file will have to be transferred manually and the cluster stack software will have to be restarted to catch up once it comes back online.
You can confirm that the configuration file was successfully
distributed by issuing the ccs_config_dump command on any of
the other cluster nodes and comparing the XML output.
[root@c3 ~]# ccs_config_dump
<?xml version="1.0"?>
<cluster config_version="3" name="hacluster">
<logging debug="off"/>
<clusternodes>
<clusternode name="c1" nodeid="1"/>
<clusternode name="c2" nodeid="2"/>
<clusternode name="c3" nodeid="3"/>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="simple" nofailback="1" ordered="0" restricted="0">
<failoverdomainnode name="c1"/>
<failoverdomainnode name="c2"/>
<failoverdomainnode name="c3"/>
</failoverdomain>
</failoverdomains>
</rm>
<cman nodename="c3" cluster_id="10117"/>
</cluster>
setenforce 0
All nodes are under oracle virtual box:-
number of cluster nodes : 3
setup static ip for all 3 nodes and proper hostname
example
[root@c1 cluster]# cat /etc/hosts
127.0.0.1 localhost
192.168.0.55 c1.example.com c1
192.168.0.66 c2.example.com c2
192.168.0.77 c3.example.com c3
Setup proper gateway( setup similar config in other 2 nodes)
[root@c1]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=c1.example.com
GATEWAY=192.168.0.235
[root@c1 ]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=08:00:27:2E:24:80
TYPE=Ethernet
UUID=02e21921-20a5-4490-bf4a-052ab8cb6c11
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPADDR=192.168.0.55
NETMASK=255.255.255.0
NTP configuration: ON ALL 3 nodes. You must install and start the NTP time synchronization
service to maintain consistent time throughout the cluster:
[root@node-1 ~]# yum install ntp –y
Installation time is synchronized with official CentOS NTP
servers. If you want to change this, edit the NTP configuration file at /etc/
ntp.conf.
You must start the NTP service and make sure it starts at boot. In the
following screenshot, you can see the commands used:
[root@c1 ~]# service ntpd start
[root@c1 ~]# chkconfig ntpd on
Setting up corosync
setup iptables to allow cluster communication traffic among cluster nodes. You must add an iptables rule to allow UDP traffic on 5404 and 5405, and another rule to allow multicast traffic communication :
iptables -I INPUT -m state --state NEW -p udp -m multiport --dports 5404,5405 -j ACCEPT
iptables -I INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
The default Corosync cluster communication ports are 5404 and
5405, and the UDP protocol is used. The ports are configured in the
/etc/corosync/corosync.conf configuration file. If you want
to change the default Corosync communication ports, make sure you
change iptables accordingly.
service iptables save
yum install corosync -y #do it on all nodes
#generate corosync encryption key on any one node and transfer it to rest of nodes
#corosync-keygen
it might take long time so to speed up
dd if=/dev/random of=test123.log
Make backup of original file and rename on any one node.
[root@c1 ~]# cp /etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
#vim corosync.conf
Remove all commented-out lines—lines starting with #—and change the
secauth parameter to on.
Change the bindnetaddr parameter to the current cluster node's IP address.
* Remove all commented-out lines—lines starting with #—and change the
secauth parameter to on.
* Change the bindnetaddr parameter to the current cluster node's IP address.
* Change the to_syslog parameter to no. We will use a separate log file, so there's no need to duplicate the information in syslog.
Now scp the corosync.conf to all rest of nodes
On all nodes
#service corosync start
#chkconfig corosync on
To check if members have joined
[root@c3 ~]# corosync-objctl | grep member.*ip
runtime.totem.pg.mrp.srp.members.3.ip=r(0) ip(192.168.0.77)
runtime.totem.pg.mrp.srp.members.1.ip=r(0) ip(192.168.0.55)
runtime.totem.pg.mrp.srp.members.2.ip=r(0) ip(192.168.0.66)
You can stop corosync for some time on all nodes
#service corosync stop
Install CMAN on all nodes
yum install cman -y
Create cluster.conf file on any one node and copy to rest of nodes
cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="3" name="hacluster">
<logging debug="off"/>
<clusternodes>
<clusternode name="c1" nodeid="1"/>
<clusternode name="c2" nodeid="2"/>
<clusternode name="c3" nodeid="3"/>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="simple" nofailback="1" ordered="0" restricted="0">
<failoverdomainnode name="c1"/>
<failoverdomainnode name="c2"/>
<failoverdomainnode name="c3"/>
</failoverdomain>
</failoverdomains>
</rm>
</cluster>
The config_version parameter is the version of the CMAN
configuration file(it should be same on all nodes and should be increased by one+ when changed similar to serial number in bind zone configuration), and the name parameter is your preferred cluster name.
The debug parameter is set to off and should be turned on for debugging purposes.The clusternodes section is where the cluster nodes are configured. The clusternode parameter is defined by a name parameter, which is the FQDN or short node name, and the nodeid parameter, which is the ID of the cluster node.
Once the CMAN service has been successfully started on all cluster nodes, check the CMAN node status using the cman_tool nodes command. All three cluster nodes should be listed with status as M. In the following screenshot, you can see the output of the cman_tool nodes command issued on the node-1 cluster node:
[root@c1 ~]# cman_tool nodes
Node Sts Inc Joined Name
1 M 364 2015-08-26 10:53:04 c1
2 M 372 2015-08-26 10:54:14 c2
3 M 368 2015-08-26 10:53:18 c3
Let's take a look at each column in more detail:
• Node: This column shows the node ID number.
• Sts: This column shows the cluster node status, with M indicating that the
node is a joined cluster member and X indicating that the node is dead.
• Inc: This column is for debugging purposes only. It is the incarnation number.
• Joined: This column shows the time the cluster member joined the cluster.
• Name: This column shows the cluster node name as defined in the cluster.conf file.
Check quorum status:-
[root@c1 ~]# cman_tool status
Version: 6.2.0
Config Version: 3
Cluster Name: hacluster
Cluster Id: 10117
Cluster Member: Yes
Cluster Generation: 372
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 7
Flags:
Ports Bound: 0
Node name: c1
Node ID: 1
Multicast addresses: 239.192.39.172
Node addresses: 192.168.0.55
Nodes: This line is where the number of your cluster nodes
should be listed
Expected votes: This line shows the number of expected votes
in the currently active configuration
Total votes: This line shows the number of total votes provided
by the CMAN configuration file
Node votes: This line shows the number of votes the current
node has
Quorum: This line shows the number of votes required to reach
Quorum
The finishing touch for the CMAN installation and configuration process is the start of a service called ricci. "The ricci service provides on-the-fly CMAN configuration file distribution."
setup password for ricci user and start ricci on all nodes-
passwd ricci
Password: ********
service ricci start
chkconfig ricci on
Setup RG Manager on all nodes:-
iptables -I INPUT -m state --state NEW -p tcp -m multiport --dports 11111,21064,41966,41967,41968,41969 -j ACCEPT
service iptables save
service iptables restart
yum install rgmanager -y
service rgmanager start
chkconfig rgmanager on
Check cluster status:-
#clustat
#Member status: This line provides information about the
Quorum status.
#Member Name: This column shows the cluster node name as
defined in the cluster.conf configuration file.
#ID: This column shows the node ID number.
#Status: This column provides information about the node's
status. The output of the clustat command must list all
the cluster nodes as online, confirming that installation and
configuration of the cluster stack software on CentOS 6 were
successful and that you did a great job at it.
Edit cluster.conf file on any one node(you may need to manually create/copypaste same file on node which is offline) , make sure to increment configversion="" after making changes:-
[root@c1 ~]# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="3" name="hacluster">
<logging debug="off"/>
<clusternodes>
<clusternode name="c1" nodeid="1"/>
<clusternode name="c2" nodeid="2"/>
<clusternode name="c3" nodeid="3"/>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="simple" nofailback="1" ordered="0" restricted="0">
<failoverdomainnode name="c1"/>
<failoverdomainnode name="c2"/>
<failoverdomainnode name="c3"/>
</failoverdomain>
</failoverdomains>
</rm>
</cluster>
To validate
[root@c1 ~]# ccs_config_validate
Configuration validates
Once it is found valid, you can propagate to other nodes
# cman_tool version -r
Note that, if a specific cluster node is not online, the configuration file will have to be transferred manually and the cluster stack software will have to be restarted to catch up once it comes back online.
You can confirm that the configuration file was successfully
distributed by issuing the ccs_config_dump command on any of
the other cluster nodes and comparing the XML output.
[root@c3 ~]# ccs_config_dump
<?xml version="1.0"?>
<cluster config_version="3" name="hacluster">
<logging debug="off"/>
<clusternodes>
<clusternode name="c1" nodeid="1"/>
<clusternode name="c2" nodeid="2"/>
<clusternode name="c3" nodeid="3"/>
</clusternodes>
<rm>
<failoverdomains>
<failoverdomain name="simple" nofailback="1" ordered="0" restricted="0">
<failoverdomainnode name="c1"/>
<failoverdomainnode name="c2"/>
<failoverdomainnode name="c3"/>
</failoverdomain>
</failoverdomains>
</rm>
<cman nodename="c3" cluster_id="10117"/>
</cluster>
No comments:
Post a Comment