Automate Apache Tomcat Clusters

The following script is a handy tool which I use to install an apache instance with mod_jk module and several tomcat instances which are used as a loadbalancing setup. After executing this script all you have to do is start the individual servers and everything should work.

The code

#!/bin/bash
################################################################################
# This script is to automate a typical apache tomcat loadbalanced setup
# Author: Tharun Kumar Allu
# Email : tharun_DOT_allu_AT_gmail.com
################################################################################
# Setup all the variables properly
# Change the source to your choice before executing the script
# The only requirement in this script is they have to be tar.gz
# if you have other formats then change the basename directive below to reflect
# that
APACHE_SOURCE="http://apache.seekmeup.com/httpd/httpd-2.2.8.tar.gz"
TOMCAT_BINARY="http://www.uniontransit.com/apache/tomcat/tomcat-6/v6.0.16/bin/apache-tomcat-6.0.16.tar.gz"
MODJK_SOURCE="http://www.apache.org/dist/tomcat/tomcat-connectors/jk/source/jk-1.2.26/tomcat-connectors-1.2.26-src.tar.gz"
# Number of Tomcat clusters that you want to be installed
NUM_CLUSTERS=2
# Apache parameters to ./configure script enable more modules you might require
# here
APACHE_CONFIG=" --enable-so --enable-jk"

# Path where all the applications are installed  if this path is not accessable
# by the user executing this script then prefix su - to 'make install' for
# apache and mod_jk
INSTALL_PREFIX="/path/to/your/install/directory"

# Set the prefix of the ports that are used by Tomcat here for example port 8080
# is changed to 12080 for cluster 0 and 13080 and so on for the other clusters

TC_PORT_PREFIX_ORIG=12

################################################################################
OLD_PWD=`pwd`

mkdir temp$$

cd temp$$

wget $APACHE_SOURCE
wget $TOMCAT_BINARY
wget $MODJK_SOURCE

tar zxvf `basename $APACHE_SOURCE` >/dev/null
cd `basename $APACHE_SOURCE .tar.gz`

./configure --prefix=$INSTALL_PREFIX/apache/ $APACHE_CONFIG
make
# prefix su - to the following command if current user does not have write
# access to $INSTALL_PREFIX
make install
cd ..
pwd

# Installing tomcat clusters and changing server.xml

TC_PORT_PREFIX=$TC_PORT_PREFIX_ORIG

for ((i=0;i<$NUM_CLUSTERS;i++));do         tar zxvf `basename $TOMCAT_BINARY` >/dev/null
       mv `basename $TOMCAT_BINARY .tar.gz` $INSTALL_PREFIX/tomcat-cluster-$i
       mv $INSTALL_PREFIX/tomcat-cluster-$i/conf/server.xml $INSTALL_PREFIX/tomcat-cluster-$i/conf/server.xml.orig
       cat $INSTALL_PREFIX/tomcat-cluster-$i/conf/server.xml.orig | sed s/"Engine name=\"Catalina\""/"Engine name=\"Catalina\" jvmRoute=\"cluster-"$i"\""/ | sed s/8005/"$TC_PORT_PREFIX"005/ | sed s/8080/"$TC_PORT_PREFIX"080/ | sed s/8009/"$TC_PORT_PREFIX"009/ >$INSTALL_PREFIX/tomcat-cluster-$i/conf/server.xml
       TC_PORT_PREFIX=`expr $TC_PORT_PREFIX + 1`;
done

# Compiling and installing mod_jk.so
tar zxvf `basename $MODJK_SOURCE`
cd `basename $MODJK_SOURCE .tar.gz`
cd native
./configure --with-apxs=$INSTALL_PREFIX/apache/bin/apxs
make
# prefix su - to the following command if current user does not have write
# access to $INSTALL_PREFIX
make install

cd $OLD_PWD

rm -rf temp$$

touch $INSTALL_PREFIX/apache/conf/workers.properties

# Adding JK related entries to apache httpd.conf
echo "JkWorkersFile \"conf/workers.properties\"" >> $INSTALL_PREFIX/apache/conf/httpd.conf
echo "JkLogFile \"logs/mod_jk.log\"" >>$INSTALL_PREFIX/apache/conf/httpd.conf

echo "JkLogLevel info" >>$INSTALL_PREFIX/apache/conf/httpd.conf

echo "JkMount /jkstatus* jkstatus" >> $INSTALL_PREFIX/apache/conf/httpd.conf

#Building the loadbalancer worker.properties file

echo "worker.list=loadbalancer,jkstatus" >> $INSTALL_PREFIX/apache/conf/workers.properties

TC_PORT_PREFIX=$TC_PORT_PREFIX_ORIG

for ((i=0;i<$NUM_CLUSTERS;i++));do         echo "worker.type=ajp13" >> $INSTALL_PREFIX/apache/conf/workers.properties
       TEMP_X=`hostname`;
       HOST=`host $TEMP_X  | awk '{print $4}'`
       echo "worker.cluster-$i.host=$HOST" >> $INSTALL_PREFIX/apache/conf/workers.properties
       echo "worker.cluster-$i.port=$TC_PORT_PREFIX"009 >> $INSTALL_PREFIX/apache/conf/workers.properties
       echo "worker.cluster-$i.lbfactor=1" >>  $INSTALL_PREFIX/apache/conf/workers.properties
       BALANCED_WORKERS=$BALANCED_WORKERS,cluster-$i
       TC_PORT_PREFIX=`expr $TC_PORT_PREFIX + 1`;
done

echo "worker.loadbalancer.balance_workers=$BALANCED_WORKERS" >> $INSTALL_PREFIX/apache/conf/workers.properties
echo "worker.jkstatus.type=status" >> $INSTALL_PREFIX/apache/conf/workers.properties

mv $INSTALL_PREFIX/apache/conf/httpd.conf $INSTALL_PREFIX/apache/conf/httpd.conf.orig

sed s/"Listen 80"/"Listen 80\n\n\nLoadModule modules\/mod_jk.so\n"/ $INSTALL_PREFIX/apache/conf/httpd.conf.orig > $INSTALL_PREFIX/apache/conf/httpd.conf

Paging / monitoring service using Perl

Here I have written a small perl program which is used to read a configuration file where you can specify different types of services to be monitored and send out an email / page (email to mobile number).

This is pager.conf example

# This configuration file is used to setup paging for individual applications
# each filed is separated by one tab only.
# FIELD1 is server name
# FIELD2 is list of comma separated processes
# FIELD3 is type of process db - database proc - process mlog - message log
# FIELD4 is email address of the pager can be comma separated for multiple
# recipients.
#=============================================
# server        processes       type    email
#=============================================
server1    db    user@domail.com

Here is the actual perl program

#!/usr/bin/perl
$CONF="/path/to/pager.conf";
open(F,"<$CONF") or die "Cannot open $CONF\n";

# Setting up default email.
$to=="user\@domain.com";
$from="pager\@server.com";
$subject="Paging default";
$out = "Default message!";

$line= < F>
while($line)
{
        chomp($line);
# Ignore Comments
        if($line =~ m/^#/)
        {
                $line= < F>
                next;
        }
#Ignore Empty lines
        if($line =~ m/^(\s)*$/)
        {
                $line= < F>
                next;
        }
# Get server proc type and email information from config file
        ($server,$proc,$type,$email)=split(/\t/,$line);
        @proclist=split(/,/,$proc);
        #print "Server=$server\nproc=$proc\ntype=$type\nemail=$email\n";
        foreach $i (@proclist)
        {
# if it is a database check whether its up or not send out email alerts if it is down
                if($type eq "db")
                {
                        $ret=system("Your Database related checking script/ command");
                        #print "return = $ret\n";
                        if( $ret != 0 )
                        {
                                $to=$email;
                                $subject="$server $i $type is down";
                                $out="Paging service Info:\n$server $i $type is down.\nsincerely,\nPager\n";
                                email_alert();
                        }
                }
                elsif($type eq "proc")
                {
# if it is a process check whether its running or not send out email alerts if it is down
                        $ret=system("Your process related script / command");
                        #print "return = $ret\n";
                        if( $ret != 0 )
                        {
                                $to=$email;
                                $subject="$server $i $type is down";
                                $out="Paging service Info:\n$server $i $type is down.\nsincerely,\nPager\n";
                                email_alert();
                        }
                }
                elsif($type eq "mlog")
                {
                        $ret=system("Your log search script / command");
                        #print "return = $ret\n";
                        if( $ret == 0 )
                        {
                                $to=$email;
                                $subject="$server $i has error";
                                $out="Paging service Info:\nOn $server $i log has error in it.\nsincerely,\nPager\n";
                                email_alert();
                        }
                }

        }
        $line= < F>
}

#Function that sends email look at the configuration for default email at the top of the script

sub email_alert
{
        #print "entered email\n";
        #print "$to\n$from\n$subject\n$out\n";
        # send email using UNIX/Linux sendmail
        open(MAIL, "|/usr/sbin/sendmail -t");

        ## Mail Header
        print MAIL "To: $to\n";
        print MAIL "From: $from\n";
        print MAIL "Subject: $subject\n";

        ## Mail Body
        print MAIL $out;
        close(MAIL);
}

This script could give you a starting point to add more services. Once it is done run it as a cron job every x amount of time where x is your interval at which your want to test for these services. On my side I test them every 5 minutes. with a cron entry .

SSH with no password

This is a very handy setting I use to login to various machines with out actually entering the password every time.
I have used this to setup user accounts on different servers so that I can monitor processes, disk space, etc.
lets first do the actual no password setup.

On Server 1
$ ssh-keygen -t dsa
Do not enter any password for key as this will defeat the whole purpose of not typing a password.
this creates id_dsa and id_dsa.pub files in ~/.ssh which are private and public keys respectively.
On Server 2
copy id_dsa.pub from Server 1 to Server 2
append the contents of id_dsa.pub to ~/.ssh/authorized_keys2
$ cat id_dsa.pub >> ~/.ssh/authorized_keys2

Now ssh from Server 1 for that user to Server 2 does not require a password.

Moving mediawiki to a different server

• make a dump of the database

ex:
mysqldump --port=2081 --socket=mysql.sock --user=root
--password=yourpasswd --databases wikidb --single-transaction
--flush-logs | bzip2 -c > backup.bz2

• make a archive of the web folder of wiki

ex: tar jcvf web.backup.tbz wiki/

• On the new server install mysql, php , php-mysql, httpd
• create a database and user in mysql.
• change LocalSettings.php to point to new database
• start/restart httpd

Apache with ssl on Solaris

You have to install openssl if it is not already installed.
$ ./configure --prefix=/path/to/apache/ --enable-so --enable-jk --enable-ssl --with-ssl=/path/to/openssl/
$ make
# make install

HTTP setup for accessing subversion

Install modules mod_dav_svn.so mod_authz_svn.so
and make the following change in your httpd.conf

      DAV svn
      SVNPath /path/to/svn
      AuthType Basic
      AuthName "Your Subversion"
      AuthUserFile /path/to/htpasswd
      Require valid-user

Note: For further security you can have this location accessible only through SSL by using virtual host directive.

How to move subversion (SVN)

• on old server svnadmin dump /repo/path > repo.dmp
• on new server
svnadmin create /repo/path and svnadmin load /repo/path < repo.dmp


Long Due

This is something which has been long due. I have been using linux from a long time and wanted to post my difficulties and achievements in a blog but did not get an opportunity to do so. Now that I have been working with Linux in an Enterprise level, I though I will put some of my ideas implementations etc so that others can easily pickup.