RCRON – Setup High Availability of cron Jobs

Overview :

rcron is a  powerful  tool that helps  system administrators  in setting up cron jobs redundancy and failover over groups of machines. RCRON ensure that a job installed on several machines will only run on the activeone at any time.

Working of RCRON …..

High Availability using RCRON ( One Node will be stamped as Active and Second Node will be stamped as Passive ) , Same cron configuration will be on both , only difference would be active/passive state in a file.

For automatic Switching of active/passive state , we will be using KEEPALIVED Daemon , which utilizes the keepalive signal for communication between 2 nodes. After a signal is sent, if no reply is received the link is assumed to be down.

In this scenario , One node is marked as KEEPALIVED master and second as KEEPALIVED backup. Master node will keep the rcron state active , and backup node will keep the rcron state as passive. As soon as the Master Node comes down , it will send a 0 priority signal to the Backup Node , which will switch over and behave as a MASTER NODE and consecutively mark rcron as active as soon as the master node comes up again , the Backup Node will switch over to backup mode .

Installation and Configuration Steps of RCRON On CentOS / RHEL

In My scenario I have Two Linux Machines , one is Primary and second one is Slave.

Step:1 Enable IP Forwarding on both the Nodes. Edit the file /etc/sysctl.conf & make the below change
net.ipv4.ip_forward = 1
# sysctl -p      ; Save the Changes without rebooting the Machine

 

Step:2 Add the EPEL Repository On Both Nodes
# wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm

 

Step:3 Install the  necessary packages on Both the Nodes
# yum install subversion byacc flex gcc

 

Step:4 If You getting internet from the proxy server , then set http_proxy for svn
# mkdir /root/.subversion/
#  vi ~/.subversion/servers
http-proxy-exceptions = *.example.com
http-proxy-host = www.example.com
http-proxy-port = 8080
http-compression = no

 

Step:5 If we are getting Internet connection directly , then skip proxy settings. Download the rcron software from SVN
# cd /root;
# svn co http://rcron.googlecode.com/svn/trunk rcron
# cd rcron/rcron
# ./configure
# make
# make install

 

Step:6 Create the directory on both the nodes
# mkdir /etc/rcron/
##############################################
For MASTER SERVER Setup RCRON.conf as follows
##############################################
#vi /etc/rcron/rcron.conf
# An arbitrary name
cluster_name        = cluster
# A file containing either the word "active" or the word "passive"
state_file          = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state       = active
syslog_facility     = LOG_CRON
syslog_level        = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level          = 19

 

+++++++++++++++++++++++++++++++++++++++++++++++
For BACKUP SERVER NODE , Setup RCRON.conf as follows
+++++++++++++++++++++++++++++++++++++++++++++++
vi /etc/rcron/rcron.conf;
# An arbitrary name
cluster_name = cluster
# A file containing either the word "active" or the word "passive"
state_file = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state = passive
syslog_facility = LOG_CRON
syslog_level = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level = 19

On the Master Node Run Below Commands :

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "active" > /var/run/rcron/state

 

On the Slave Node Run the Below Commands :

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "passive" > /var/run/rcron/state
Step:7 Now install the keepalive package on both the nodes

a) Download the latest source tarball

# cd /root
# wget http://cgit.luffy.cx/keepalived/snapshot/keepalived-1.2.7.tar.gz

 

b) Install the RPM BUILD package

# yum -y install rpm-build

 

c) Untar the tar ball & edit keepalived.spec.in file

# tar -zxvf keepalived-1.2.7.tar.gz
# mkdir -p /root/rpmbuild/SOURCES/
# cp /root/keepalived-1.2.7.tar.gz /root/rpmbuild/SOURCES/
# vi /root/keepalived-1.2.7/keepalived.spec.in
replace from Version 1.2.2 to Version: 1.2.7
# yum -y install popt*
# cd /root/keepalived-1.2.7
# rpmbuild -ba keepalived.spec.in

Above Command will create a compiled RPM from Source in the location :

/root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

Now Install rpm with rpm command :

# rpm -ivh /root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

 

Step:8 SETUP the keepalived configuration

++++++++++++++++++++++++++++++++++++++++++++

For the master node : setup the following

++++++++++++++++++++++++++++++++++++++++++++

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 { 
state MASTER 
interface eth0 
lvs_sync_daemon_inteface eth0 
virtual_router_id 31 
priority 101 
advert_int 5
vrrp_unicast_bind 172.16.243.144 
vrrp_unicast_peer 172.16.243.145 
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

 

+++++++++++++++++++++++++++++++++++++++++++++++

For the backup node :

+++++++++++++++++++++++++++++++++++++++++++++++

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state BACKUP 
interface eth0 
lvs_sync_daemon_inteface eth0
virtual_router_id 31
priority 100 
advert_int 5 
vrrp_unicast_bind 172.16.243.145 
vrrp_unicast_peer 172.16.243.144
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

Where “172.16.243.145” and “172.16.243.144” are the IP address of Linux Machines

Step:9 Create Sample crontab entry on both master and slave.

* * * * * /usr/local/bin/rcron --conf /etc/rcron/rcron.conf echo `date` >> /tmp/output

Start the Keepalive service on both the nodes

# service keepalived start ; chkconfig keepalived on

Note : - In case the keepalived daemon gets killed we are going to create a crontab script

* * * * * * /root/keep-alive-monitor

Contents of the script

++++++++++++++++++++++++++++++++++++++++++

cat /root/keep-alive-monitor
#!/bin/sh
echo "test" >> /tmp/monitor;
ps -ef|grep -v grep|grep -i keepalived;
if [ $? -eq 0 ] ; then 
exit 0
else 
echo "passive" > /var/run/rcron/state;
/etc/init.d/keepalived restart;
fi

++++++++++++++++++++++++++++++++++++++++++

Leave a Reply

Your email address will not be published. Required fields are marked *