RCRON – Setup High Availability of cron Jobs

Overview :

rcron is a  powerful  tool that helps  system administrators  in setting up cron jobs redundancy and failover over groups of machines. RCRON ensure that a job installed on several machines will only run on the activeone at any time.

Working of RCRON …..

High Availability using RCRON ( One Node will be stamped as Active and Second Node will be stamped as Passive ) , Same cron configuration will be on both , only difference would be active/passive state in a file.

For automatic Switching of active/passive state , we will be using KEEPALIVED Daemon , which utilizes the keepalive signal for communication between 2 nodes. After a signal is sent, if no reply is received the link is assumed to be down.

In this scenario , One node is marked as KEEPALIVED master and second as KEEPALIVED backup. Master node will keep the rcron state active , and backup node will keep the rcron state as passive. As soon as the Master Node comes down , it will send a 0 priority signal to the Backup Node , which will switch over and behave as a MASTER NODE and consecutively mark rcron as active as soon as the master node comes up again , the Backup Node will switch over to backup mode .

Installation and Configuration Steps of RCRON On CentOS / RHEL

In My scenario I have Two Linux Machines , one is Primary and second one is Slave.

Step:1 Enable IP Forwarding on both the Nodes. Edit the file /etc/sysctl.conf & make the below change

net.ipv4.ip_forward = 1
# sysctl -p      ; Save the Changes without rebooting the Machine

Step:2 Add the EPEL Repository On Both Nodes

# wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm

Step:3 Install the  necessary packages on Both the Nodes

# yum install subversion byacc flex gcc

Step:4 If You getting internet from the proxy server , then set http_proxy for svn

# mkdir /root/.subversion/
#  vi ~/.subversion/servers
http-proxy-exceptions = *.example.com
http-proxy-host = www.example.com
http-proxy-port = 8080
http-compression = no

Step:5 If we are getting Internet connection directly , then skip proxy settings. Download the rcron software from SVN

# cd /root;
# svn co http://rcron.googlecode.com/svn/trunk rcron
# cd rcron/rcron
# ./configure
# make
# make install

Step:6 Create the directory on both the nodes

# mkdir /etc/rcron/
##############################################
For MASTER SERVER Setup RCRON.conf as follows
##############################################
#vi /etc/rcron/rcron.conf
# An arbitrary name
cluster_name        = cluster
# A file containing either the word "active" or the word "passive"
state_file          = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state       = active
syslog_facility     = LOG_CRON
syslog_level        = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level          = 19

+++++++++++++++++++++++++++++++++++++++++++++++
For BACKUP SERVER NODE , Setup RCRON.conf as follows
+++++++++++++++++++++++++++++++++++++++++++++++
vi /etc/rcron/rcron.conf;
# An arbitrary name
cluster_name = cluster
# A file containing either the word "active" or the word "passive"
state_file = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state = passive
syslog_facility = LOG_CRON
syslog_level = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level = 19

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "active" > /var/run/rcron/state

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "passive" > /var/run/rcron/state

# cd /root
# wget http://cgit.luffy.cx/keepalived/snapshot/keepalived-1.2.7.tar.gz

# yum -y install rpm-build

# tar -zxvf keepalived-1.2.7.tar.gz
# mkdir -p /root/rpmbuild/SOURCES/
# cp /root/keepalived-1.2.7.tar.gz /root/rpmbuild/SOURCES/
# vi /root/keepalived-1.2.7/keepalived.spec.in
replace from Version 1.2.2 to Version: 1.2.7
# yum -y install popt*
# cd /root/keepalived-1.2.7
# rpmbuild -ba keepalived.spec.in

# rpm -ivh /root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 { 
state MASTER 
interface eth0 
lvs_sync_daemon_inteface eth0 
virtual_router_id 31 
priority 101 
advert_int 5
vrrp_unicast_bind 172.16.243.144 
vrrp_unicast_peer 172.16.243.145 
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state BACKUP 
interface eth0 
lvs_sync_daemon_inteface eth0
virtual_router_id 31
priority 100 
advert_int 5 
vrrp_unicast_bind 172.16.243.145 
vrrp_unicast_peer 172.16.243.144
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

* * * * * /usr/local/bin/rcron --conf /etc/rcron/rcron.conf echo `date` >> /tmp/output

# service keepalived start ; chkconfig keepalived on

* * * * * * /root/keep-alive-monitor

cat /root/keep-alive-monitor
#!/bin/sh
echo "test" >> /tmp/monitor;
ps -ef|grep -v grep|grep -i keepalived;
if [ $? -eq 0 ] ; then 
exit 0
else 
echo "passive" > /var/run/rcron/state;
/etc/init.d/keepalived restart;
fi