RCRON – Setup High Availability of cron Jobs

Overview :

rcron is a  powerful  tool that helps  system administrators  in setting up cron jobs redundancy and failover over groups of machines. RCRON ensure that a job installed on several machines will only run on the activeone at any time.

Working of RCRON …..

High Availability using RCRON ( One Node will be stamped as Active and Second Node will be stamped as Passive ) , Same cron configuration will be on both , only difference would be active/passive state in a file.

For automatic Switching of active/passive state , we will be using KEEPALIVED Daemon , which utilizes the keepalive signal for communication between 2 nodes. After a signal is sent, if no reply is received the link is assumed to be down.

In this scenario , One node is marked as KEEPALIVED master and second as KEEPALIVED backup. Master node will keep the rcron state active , and backup node will keep the rcron state as passive. As soon as the Master Node comes down , it will send a 0 priority signal to the Backup Node , which will switch over and behave as a MASTER NODE and consecutively mark rcron as active as soon as the master node comes up again , the Backup Node will switch over to backup mode .

Installation and Configuration Steps of RCRON On CentOS / RHEL

In My scenario I have Two Linux Machines , one is Primary and second one is Slave.

Step:1 Enable IP Forwarding on both the Nodes. Edit the file /etc/sysctl.conf & make the below change
net.ipv4.ip_forward = 1
# sysctl -p      ; Save the Changes without rebooting the Machine

 

Step:2 Add the EPEL Repository On Both Nodes
# wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
# rpm -ivh epel-release-6-8.noarch.rpm

 

Step:3 Install the  necessary packages on Both the Nodes
# yum install subversion byacc flex gcc

 

Step:4 If You getting internet from the proxy server , then set http_proxy for svn
# mkdir /root/.subversion/
#  vi ~/.subversion/servers
http-proxy-exceptions = *.example.com
http-proxy-host = www.example.com
http-proxy-port = 8080
http-compression = no

 

Step:5 If we are getting Internet connection directly , then skip proxy settings. Download the rcron software from SVN
# cd /root;
# svn co http://rcron.googlecode.com/svn/trunk rcron
# cd rcron/rcron
# ./configure
# make
# make install

 

Step:6 Create the directory on both the nodes
# mkdir /etc/rcron/
##############################################
For MASTER SERVER Setup RCRON.conf as follows
##############################################
#vi /etc/rcron/rcron.conf
# An arbitrary name
cluster_name        = cluster
# A file containing either the word "active" or the word "passive"
state_file          = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state       = active
syslog_facility     = LOG_CRON
syslog_level        = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level          = 19

 

+++++++++++++++++++++++++++++++++++++++++++++++
For BACKUP SERVER NODE , Setup RCRON.conf as follows
+++++++++++++++++++++++++++++++++++++++++++++++
vi /etc/rcron/rcron.conf;
# An arbitrary name
cluster_name = cluster
# A file containing either the word "active" or the word "passive"
state_file = /var/run/rcron/state
# The default state in case state_file can't be read
#default_state = passive
syslog_facility = LOG_CRON
syslog_level = LOG_INFO
# We can tune jobs niceness/priorities 
nice_level = 19

On the Master Node Run Below Commands :

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "active" > /var/run/rcron/state

 

On the Slave Node Run the Below Commands :

# mkdir /var/run/rcron
# touch /var/run/rcron/state
# echo "passive" > /var/run/rcron/state
Step:7 Now install the keepalive package on both the nodes

a) Download the latest source tarball

# cd /root
# wget http://cgit.luffy.cx/keepalived/snapshot/keepalived-1.2.7.tar.gz

 

b) Install the RPM BUILD package

# yum -y install rpm-build

 

c) Untar the tar ball & edit keepalived.spec.in file

# tar -zxvf keepalived-1.2.7.tar.gz
# mkdir -p /root/rpmbuild/SOURCES/
# cp /root/keepalived-1.2.7.tar.gz /root/rpmbuild/SOURCES/
# vi /root/keepalived-1.2.7/keepalived.spec.in
replace from Version 1.2.2 to Version: 1.2.7
# yum -y install popt*
# cd /root/keepalived-1.2.7
# rpmbuild -ba keepalived.spec.in

Above Command will create a compiled RPM from Source in the location :

/root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

Now Install rpm with rpm command :

# rpm -ivh /root/rpmbuild/RPMS/x86_64/keepalived-1.2.7-5.x86_64.rpm

 

Step:8 SETUP the keepalived configuration

++++++++++++++++++++++++++++++++++++++++++++

For the master node : setup the following

++++++++++++++++++++++++++++++++++++++++++++

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 { 
state MASTER 
interface eth0 
lvs_sync_daemon_inteface eth0 
virtual_router_id 31 
priority 101 
advert_int 5
vrrp_unicast_bind 172.16.243.144 
vrrp_unicast_peer 172.16.243.145 
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

 

+++++++++++++++++++++++++++++++++++++++++++++++

For the backup node :

+++++++++++++++++++++++++++++++++++++++++++++++

# vi /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state BACKUP 
interface eth0 
lvs_sync_daemon_inteface eth0
virtual_router_id 31
priority 100 
advert_int 5 
vrrp_unicast_bind 172.16.243.145 
vrrp_unicast_peer 172.16.243.144
authentication { 
auth_type PASS 
auth_pass 1111
}
notify_backup "/bin/echo passive > /var/run/rcron/state" 
notify_master "/bin/echo active > /var/run/rcron/state" 
notify_fault "/bin/echo passive > /var/run/rcron/state"
}

Where “172.16.243.145” and “172.16.243.144” are the IP address of Linux Machines

Step:9 Create Sample crontab entry on both master and slave.

* * * * * /usr/local/bin/rcron --conf /etc/rcron/rcron.conf echo `date` >> /tmp/output

Start the Keepalive service on both the nodes

# service keepalived start ; chkconfig keepalived on

Note : – In case the keepalived daemon gets killed we are going to create a crontab script

* * * * * * /root/keep-alive-monitor

Contents of the script

++++++++++++++++++++++++++++++++++++++++++

cat /root/keep-alive-monitor
#!/bin/sh
echo "test" >> /tmp/monitor;
ps -ef|grep -v grep|grep -i keepalived;
if [ $? -eq 0 ] ; then 
exit 0
else 
echo "passive" > /var/run/rcron/state;
/etc/init.d/keepalived restart;
fi

++++++++++++++++++++++++++++++++++++++++++

Free Download Linux e-Books

2 comments

  1. Jhonathan says:

    Hello

    Excellent tutorial. Right to the point. I was just wondering if there is a way I can test the keepAlive part. I can’t shutdown my server, so I changing the IP address for fake ones in the keepAlive scrip does’t work or shutting down the KeepAlive service on primary doesn’t make secondary to wake up.

    Thank you

  2. Marius says:

    Hi there!

    Thanks! for the tutorial, very helpful. Everything works fine with the exception of changing the state when the Primary server is down. My both servers (Ubuntu 12.04) are in different Azure clouds and I can connect by ssh one to each other.

    I’ve tried many changes but couldn’t get it to work. The bind in keepalived.conf is the local ip/hostname and the peer the the remote server. I shutdown the Master and the Slave keeps being passive. Any ideas?

    Thank you!

Leave a Reply

Your email address will not be published. Required fields are marked *