TOP - The Monitoring Tool

Top command provides a real-time look at what is happening with your system. Top produces so much output that a new user may get over whelmed with all thats presented and what it means.

Lets take a look at TOP one line at a time. The server has been flooded with http requests to create some load on the server.

top output:

top - 22:09:08 up 14 min,  1 user,  load average: 0.21, 0.23, 0.30
Tasks:  81 total,   1 running,  80 sleeping,   0 stopped,   0 zombie
Cpu(s):  9.5%us, 31.2%sy,  0.0%ni, 27.0%id,  7.6%wa,  1.0%hi, 23.7%si,  0.0%st
Mem:    255592k total,   167568k used,    88024k free,    25068k buffers
Swap:   524280k total,        0k used,   524280k free,    85724k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3166 apache    15   0 29444 6112 1524 S  6.6  2.4   0:00.79 httpd
 3161 apache    15   0 29444 6112 1524 S  5.9  2.4   0:00.79 httpd
 3164 apache    15   0 29444 6112 1524 S  5.9  2.4   0:00.75 httpd
 3169 apache    15   0 29444 6112 1524 S  5.9  2.4   0:00.74 httpd
 3163 apache    15   0 29444 6112 1524 S  5.6  2.4   0:00.76 httpd
 3165 apache    15   0 29444 6112 1524 S  5.6  2.4   0:00.77 httpd
 3167 apache    15   0 29444 6112 1524 S  5.3  2.4   0:00.73 httpd
 3162 apache    15   0 29444 6112 1524 S  5.0  2.4   0:00.77 httpd
 3407 root      16   0  2188 1012  816 R  1.7  0.4   0:00.51 top
  240 root      15   0     0    0    0 S  0.3  0.0   0:00.08 pdflush
  501 root      10  -5     0    0    0 S  0.3  0.0   0:01.20 kjournald
 2794 root      18   0 12720 1268  560 S  0.3  0.5   0:00.73 pcscd
    1 root      15   0  2060  636  544 S  0.0  0.2   0:03.81 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.07 events/0

The first line in top:

top - 22:09:08 up 14 min,  1 user,  load average: 0.21, 0.23, 0.30

“22:09:08″ is the current time; “up 14 min” shows how long the system has been up for; “1 user” how many users are logged in; “load average: 0.21, 0.23, 0.30″ the load average of the system (1minute, 5 minutes, 15 minutes).

Load average is an extensive topic and to understand its inner workings can be daunting. The simplest of definitions states that load average is the cpu utilization over a period of time. A load average of 1 means your cpu is being fully utilized and processes are not having to wait to use a CPU. A load average above 1 indicates that processes need to wait and your system will be less responsive. If your load average is consistently above 3 and your system is running slow you may want to upgrade to more CPU’s or a faster CPU.

The second line in top:

Tasks:  82 total,   1 running,  81 sleeping,   0 stopped,   0 zombie

Shows the number of processes and their current state.

The third lin in top:

Cpu(s):  9.5%us, 31.2%sy,  0.0%ni, 27.0%id,  7.6%wa,  1.0%hi, 23.7%si,  0.0%st

Shows CPU utilization details. “9.5%us” user processes are using 9.5%; “31.2%sy” system processes are using 31.2%; “27.0%id” percentage of available cpu; “7.6%wa” time CPU is waiting for IO.

When first analyzing the Cpu(s) line in top look at the %id to see how much cpu is available. If %id is low then focus on %us, %sy, and %wa to determine what is using the CPU.

The fourth and fifth lines in top:

Mem:    255592k total,   167568k used,    88024k free,    25068k buffers
Swap:   524280k total,        0k used,   524280k free,    85724k cached

Describes the memory usage. These numbers can be misleading. “255592k total” is total memory in the system; “167568K used” is the part of the RAM that currently contains information; “88024k free” is the part of RAM that contains no information; “25068K buffers and 85724k cached” is the buffered and cached data for IO.

So what is the actual amount of free RAM available for programs to use ?

The answer is: free + (buffers + cached)

88024k + (25068k + 85724k) = 198816k

How much RAM is being used by progams ?

The answer is: used – (buffers + cached)

167568k – (25068k + 85724k) = 56776k

The processes information:

Top will display the process using the most CPU usage in descending order. Lets describe each column that represents a process.

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
3166 apache    15   0 29444 6112 1524 S  6.6  2.4   0:00.79 httpd

PID – process ID of the process

USER – User who is running the process

PR – The priority of the process

NI – Nice value of the process (higher value indicates lower priority)

VIRT – The total amount of virtual memory used

RES – Resident task size

SHR – Amount of shared memory used

S – State of the task. Values are S (sleeping), D (uninterruptible sleep), R (running), Z(zombies), or T (stopped or traced)

%CPU – Percentage of CPU used

%MEM – Percentage of Memory used

TIME+ – Total CPU time used

COMMAND – Command issued

Interacting with TOP

Now that we are able to understand the output from TOP lets learn how to change the way the output is displayed.

Just press the following key while running top and the output will be sorted in real time.

M – Sort by memory usage

P – Sort by CPU usage

T – Sort by cumulative time

z – Color display

k – Kill a process

q – quit

If we want to kill the process with PID 3161, then press “k” and a prompt will ask you for the PID number, and enter 3161.

Command Line Parameters with TOP

You can control what top displays by issuing parameters when you run top.

- d – Controls the delay between refreshes

- p – Specify the process by PID that you want to monitor

-n – Update the display this number of times and then exit

If we want to only monitor the http process with a PID of 3166

$ top -p 3166

If we want to change the delay between refreshes to 5 seconds

$ top -d 5

Understanding Hard Disk Partitions

What is a hard disk partition

When a hard drive is installed in a computer, it must be partitioned before you can format and use it. Partitioning a drive is when you divide the total storage of a drive into different pieces. These pieces are called partitions. Once a partition is created, it can then be formatted so that it can be used on a computer. When partitions are made, you specify the total amount of storage that you would like to allocate to that partition from the total size of the drive. For example, if you have an 80 GB drive, then it would be possible to make one partition consisting of the entire 80 GB of available storage. Alternatively,you could make two partitions consisting of a 20 GB partition that will be used for the operating system and programs and a 60 GB partition set aside for data, music, and images.

Why make partitions

Now that you know what a partition is, you may be wondering why you would even need to make multiple partitions instead of just making one. Though there are quite a few reasons, we will touch on some of the more important ones below:

Multiple Filesystems - When you create a partition, you then need to format it before it can be used. When formatting a partition you tell the operating system what filesystem you would like that partition to be formatted in. Once you format a partition, and therefore assign the type of filesystem you want to use with it, you can not change that filesystem without formatting the partition again and losing all the data contained on it. If you require different filesystems on your computer, whether it be for security or for a specific application, you can then make multiple partition and assign one type of filesystem to one of the partitions and another to one of the other partitions.
Partition Size - On older operating systems the maximum size of a partition was limited. Therefore if you had more storage space on a hard drive than was usable in a single partition, you would need to create other partitions to utilize this unused space.
Multiple Operating Systems - Some operating systems use filesystems that are not compatible with other operating systems. If you want to use two operating systems on the same computer that are not compatible in this way, you could then make two partitions, one for each operating system. Then a boot loader could be used to switch between them when you start your computer.
Wasted disk Space - When a filesystems store data in a partition, the larger the partition, the greater the chance of wasted space. By having multiple partitions of smaller size, you reduce the amount of waste that filesystems may create.
Seperate system files from users files - Some components of an operating system require storage space to operate. For example, in Windows if you have no hard drive space available on the drive that Windows is installed on, you will not be able to print. By creating a partition solely devoted to the operating system and then another partition dedicated to your own data and programs, there is less of a chance of you using up the space on the operating system partition with your data, and thus not causing problems.

Why only four primary partitions?

In the current IBM PC architecture, there is a partition table in the drive's Master Boot Record (section of the hard drive that contains the commands necessary to start the operating system), or MBR, that lists information about the partitions on the hard drive. This partition table is then further split into 4 partition table entries, with each entries corresponding to a partition. Due to this it is only possible to have four partitions. These 4 partitions are typically known as primary partitions. To overcome this restriction, system developers decided to add a new type of partition called the extended partition. By replacing one of the four primary partitions with an extended partition, you can then make an additional 24 logical partitions within the extended one. The table below illustrates this.

Partition Table
Primary Partition #1
Primary Partition #2
Primary Partition #3
Primary Partition #4 (Extended Partition)
  Logical Partition #1
  Logical Partition #1

As you can see, this partition table is broken up into 4 primary partitions. The fourth partition, though, has been flagged as an extended partition. This allows us to make more logical partitions under that extended partition and therefore bypassing the 4 partition limit.

Each hard drive also has one of its possible 4 partitions flagged as an active partition. The active partition is a special flag assigned to only one partition on a hard drive that the Master Boot Record (MBR) uses to boot your computer into an operating system. As only one partition may be set as the active partition, you may be wondering how people can have multiple operating systems installed on different partitions, and yet still be able to use them all. This is accomplished by installing a boot loader in the active partition. When the computer starts, it will read the MBR and determine the partition that is flagged as active. This partition is the one that contains the boot loader. When the operating system boots off of this partition the boot loader will start and allow you to choose which operating systems you would like to boot from.

Linux Basic Commands

$df -h -> same as my computer in windows

$fdsik -l -> list partition
$man <cmd> -> manual
$clear -> clear the screen
$^l -> clear the screen
$ls -> list content
$ls -l -> list content in long listing format
$ls -al -> list all subcontent in long listing format
$ll -> an alias for the above
$ls -R -> list content recursively
$l. -> list hidden files
$ls -F -> list content and classify them
$alias -> display all aliases for current user
$alias <statement> -> make alias eg alias c='clear'
$unalias <alias> -> remove alias eg unalias c
$exit -> log out from the system
$logout -> log out from the system
$^d -> log out from the system
$tree -> list content in a tree (hierarchial) diagram
$tree -d -> list subdirectories only - no files
$tree -p -> list content with their permissions
$cd <directory> -> change directory to...
$cd .. -> change to parent directory
$cd - -> change to previous directory
$cd -> change to home directory
$cd ~ -> change to home directory
$pushd -> change dir with pwd
$cat -> display a content of a file
$pwd -> print work (current) directory
$pwd -P -> print parent working dir of this symlink dir
$mkdir <directory> -> make directory
$mkdir -p <directory> -> make parent directories also if it does not exist
$touch -> make a 0 byte file if it does not exist
$cp -> copy (for files)
$cp -a -> copy (for directories)
$cp -p -> copy and preserve date and time
$mv -> move OR rename
$rmdir -> remove empty directory
$rm -> remove (for files)
$rm -f -> remove forcefully ( " " )
$rm -r -> remove recursively (for directories)
$rm -rf -> remove recursively and forcefully ( " " )
$cat -> display content of the file
$cat -n -> display content of the file and number the lines
$cal -> display calendar for current month
$date -> display system date and time
$date -s '<value>' -> change system date and time in mm/dd/yy
$hwclock -> display the hardware clock
$hwclock –hctosys -> set the system time from the hardware clock
$ln -s -> make a soft/sym/symbolic link
$ln -> make a hard link
$history -> display the list of the last 1000 commands
$! 100 -> Run command 100 in history
$vi -> text editor
$vimtutor -> vi manual withexercise
$pico -> pico manual withexercise
$mcedit -> mcedit manual withexercise
$aspell -c <filename> -> check the spelling in the file
$elinks -> check the web links
$file -> display the type of file
$which -> display the path of the binary
$whereis -> display all paths
$hostname -> display system name with domain
$id -> display id info of current user
$id -u -> display user id of current user
$id -un -> display username of current user
$id -g -> display group id of current user
$id -gn -> display groupname of current user
$uptime -> display for how long the system has been running
$tty -> display current terminal number
$users -> display no. of users currently logged in
$whoami -> display username of current user
$who -> display users logged in the system with their respective terminals and time since logged in
$who am I -> display current user, terminal and uptime
$w -> display is details which files are open on which terminal
$mkdir -p /opt/funny/test
$cd /opt/funny/test -> absolute path
$cd /opt/funny
$pwd /opt/funny
$cd test –> relative path

Monitor user activity using sudo and sudosh2

Yesterday one of my friend called up, he has a requirement at his firm. Thanks to him that he considered me asking his linux doubts. 
His requirement is as below...
He is in software development firm. They have Linux Boxes and his team is working on web application. He has created one master user say 'ALEXANDER'. This user is very important. This user posses all SUPERUSER rights. Means ALEXANDER can do anything on the system and this is the user they using for starting and shutting down their web application. Now my friend belongs to four member team and everybody has the password of ALEXANDER.
Which means if someone screwed up with project, no one would know who did it. Then the Blame Dance..

Everybody will blame each other. 
So he wants to setup his linux box such a way that whatever his team do in system, that should get logged in file.

To achieve this, there are numerous ways however I will prefer the simple one below...

So we have user ALEXANDER and four developers say SURESH,NARESH,YOGESH and NAGESH.

Note: I used UBUNTU 12.04TLS 64Bit machine to test this.

1. Change the password of ALEXANDER and do not share with anyone.

2. Now download sudosh2-1.0.4.tgz and install as instructed below

# tar zxvf sudosh2-1.0.4.tgz

# cd sudosh2-1.0.4

# CFLAGS="-D_GNU_SOURCE" ./configure

# make

# sudo make install

3. Create four users SURESH,NARESH,YOGESH and NAGESH
   
# sudo adduser SURESH
# sudo adduser NARESH
# sudo adduser YOGESH
# sudo adduser NAGESH

4. Create one GROUP say DEVELOPERS

# sudo groupadd DEVELOPERS

5. Add users SURESH,NARESH,YOGESH and NAGESH to Secondary Group DEVELOPERS

#sudo usermod -a -G DEVELOPERS SURESH
#sudo usermod -a -G DEVELOPERS NARESH
#sudo usermod -a -G DEVELOPERS YOGESH
#sudo usermod -a -G DEVELOPERS NAGESH

6. Open /etc/sudoers file and make below changes

Defaults        env_reset
Defaults        syslog=auth
Defaults>root   !set_logname
Defaults        log_year, logfile=/var/log/sudo.log
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

Cmnd_Alias SUDOSH = /usr/local/bin/sudosh

root    ALL=(ALL:ALL) ALL

%DEVELOPERS  ALL=(ALEXANDER) SUDOSH

%sudo   ALL=(ALL:ALL) ALL

This will enable the sudo log which can be found at /var/log/sudo.log
Allowing users /usr/local/bin/sudosh command only to use with ALEXANDER user.

Note: Be very careful with this file especially when you are in UBUNTU system. If you mess up with this file, there is no other option than rebooting the system and booting it in recovery mode to fix the file if you dont have policykit-1 installed.For safer side please install...

# sudo apt-get install policykit-1

7. That's it done! We are good to test this...

Login as SURESH or any one from DEVELOPERS group.

Once you get the prompt Enter..

# sudo -u ALEXANDER sudosh

will be asked for the SURESH passwd. provide it and you are in.

check with id command you should see ALEXANDER details.

# id  

uid=1000(ALEXANDER) gid=1000(ALEXANDER) groups=1000(ALEXANDER),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),111(lpadmin),112(sambashare)

You will find the logs at below location

Open a new session and login as ALEXANDER or a user who belongs to sudo group.

# cd /var/log/sudosh

# sudo ls -ltrh

-rw------- 1 ALEXANDER ALEXANDER   81 Oct 12 10:33 SURESH-ALEXANDER-time-1350063234-YTvv0lB4sLtImfI3

-rw------- 1 ALEXANDER ALEXANDER  196 Oct 12 10:33 SURESH-ALEXANDER-script-1350063234-YTvv0lB4sLtImfI3

Now you got two options to see this file either you use

# tailf SURESH-ALEXANDER-script-1350063234-YTvv0lB4sLtImfI3

you will get all details that to live if SURESH is still logged into ALEXANDER.

or use

# sudo sudosh-replay SURESH-ALEXANDER-script-1350063234-YTvv0lB4sLtImfI3 1 2

You will see Action Replay!

Linux Cache Clearing

Clearning the Linux Memory cache can be a quick way to regain system resources. 
Writing to the drop_cache process will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.


To free pagecache:

# echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:

# echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:

# echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation, and dirty objects are not freeable, the user should run “sync” command first in order to make sure all cached objects are freed.

Example – 
Memory before:

[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          1010        361        648          0         25        271
-/+ buffers/cache:         64        945
Swap:         2047          0       2047

Memory after:

[root@localhost ~]# sync

[root@localhost ~]# echo 3 > /proc/sys/vm/drop_caches

[root@localhost ~]# free -m

             total       used       free     shared    buffers     cached

Mem:          1010         81        929          0          0         20

-/+ buffers/cache:         60        950

Swap:         2047          0       2047

Apache Tomcat Server

Index

• Installation
• Setup First Instance
• Setup WEB application
• Setup Java Servlet for WEB application
• Setup Second Instance
• Setup WEB application
• Setup Java Servlet for WEB application
• Tomcat Cluster(Load Balance)using mod_jk

Today,I am going to discuss about the tomcat server with examples.Whatever I understood so far will be covered here.
I am going to install tomcat server,then will create multiple instance of it and then will deploy the web contents and so on..
Whatever I have/will learned/learn will put it here.

Installation

Installing JRE
To run Tomcat, you need Java Standard Edition (Java SE), also known as the JDK.You can download SUN's latest Java JDKs at: http://java.sun.com/javase/downloads/index.jsp
On RHEL Server type below command

#yum install java*

For my 64-bit RHEL box I downloaded the 64-bit JDK multiplatform binary for Linux: jdk-6u10-linux-x64.bin. 
I downloaded the binary file to /tmp and installed it as follows as root:
# mkdir -p /usr/java
# cd /usr/java
#
# chmod 700 /tmp/jdk-6u10-linux-x64.bin
# /tmp/jdk-6u10-linux-x64.bin
...
   creating: jdk1.6.0_10/
   creating: jdk1.6.0_10/db/
   creating: jdk1.6.0_10/db/bin/
  inflating: jdk1.6.0_10/db/bin/ij   
  inflating: jdk1.6.0_10/db/bin/NetworkServerControl  
  inflating: jdk1.6.0_10/db/bin/setNetworkClientCP.bat  
  inflating: jdk1.6.0_10/db/bin/derby_common.sh  
...
Done.
# export JAVA_HOME=/usr/java/jdk1.6.0_10
# export PATH=$JAVA_HOME/bin:$PATH
#
# which java
/usr/java/jdk1.6.0_10/bin/java
# java -version
java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) 64-Bit Server VM (build 11.0-b15, mixed mode)

Installing TOMCAT

Download the latest Tomcat 7.x version from http://tomcat.apache.org/download-70.cgi. I downloaded the Binary Core Distribution file apache-tomcat-7.0.30.tar.gz - the latest version at this time.

Installing Tomcat from a binary release (tar file) requires manual creation of the Tomcat user account. This is not necessary if you install the Tomcat RPM package on RHEL.

For security reasons I created a user account with no login shell for running the Tomcat server:

# groupadd alexander
# useradd -g alexander -s /sbin/nologin -m -d /home/alex alex

Next I extracted the tar file to /opt/tomcat and changed the ownership of all files and directories to tomcat:

# mkdir /opt/tomcat
# cd /opt/tomcat
# tar zxvf /tmp/apache-tomcat-7.0.30.tar.gz
# chown -R alex:alexander /opt/tomcat/apache-tomcat-7.0.30

The get the Tomcat version of the newly installed Tomcat, run:

# /opt/tomcat/apache-tomcat-7.0.30/bin/version.sh
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-7.0.30/temp
Using JRE_HOME:       /usr
Server version: Apache Tomcat/6.0.18
Server built:   Jul 22 2008 02:00:36
Server number:  6.0.18.0
OS Name:        Linux
OS Version:     2.6.18-6-amd64
Architecture:   x86_64
JVM Version:    1.4.2
JVM Vendor:     Free Software Foundation, Inc.

Starting/Stopping Tomcat

Now startup the Tomcat server to see whether the default Tomcat home page is being displayed. 

For security reasons I don't run the Tomcat server as user root but as alex which was created with no login shell. 
Therefore, to run Tomcat use the su command with the -p option to preserves all the environment variables when switching to alex.
And since the alex account has no login shell, it needs to be specified with the -s option. 
(You may want to use this su command if you plan on writing and implementing a system startup and shutdown script for system reboots.)

# export JAVA_HOME=/usr/java/jdk1.6.0_10
# export PATH=$JAVA_HOME/bin:$PATH
# export CATALINA_HOME=/opt/tomcat/apache-tomcat-7.0.30
# export CATALINA_BASE=/opt/tomcat/apache-tomcat-7.0.30

# su -p -s /bin/sh alex $CATALINA_HOME/bin/startup.sh
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-7.0.30/temp
Using JRE_HOME:       /usr/java/jdk1.6.0_10


Now try opening the URL http://localhost:8080 (default port 8080) from same servers browser. 
Note that you should also be able to use the name of your server instead of localhost. 
Once you opened the URL in your browser you should see Tomcat's Congratulation page. If you don't see the page, check the log files under $CATALINA_HOME/logs (/opt/tomcat/apache-tomcat-7.0.30/logs). 

To shutdown use below command,

# su -p -s /bin/sh alex $CATALINA_HOME/bin/shutdown.sh
Using CATALINA_BASE:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /opt/tomcat/apache-tomcat-7.0.30/temp
Using JRE_HOME:       /usr/java/jdk1.6.0_10

Switching to nologin shell alex User Account

Most of the next steps in this article assume that you switched to the alex user account. If you see a '$' prompt, then the steps in this article are executed as the alex user. 
If you see a '#' prompt, then the steps are executed as root. 

# su - -s /bin/sh alex

confirm with the id command,

$ id
uid=1001(alex) gid=1001(alexander) groups=1001(alexander)


Note that non-root users cannot switch to the alex account. 

Installation Complete!

Setup First Tomcat Instance

It is recommended not to store the web applications's files in Tomcat's distribution directory tree. 
For example, having a separate directory makes Tomcat upgrades easier since it won't overwrite configuration files like server.xml. 
And I am goinf to learn how to run two Tomcat instances concurrently on a single Linux server, two separate directories are needed anyway. 
I should keep in mind that it's also possible to run multiple web applications per Tomcat instance. 
I am going to create and configure one web application for each Tomcat instance. 

Setting up Directories and Files

In the following example I setup the first Tomcat instance under the base directory /opt/tomcat-instance/tomcat1.example.com. 
It's a good practice to name the base directory after the site name, in this example tomcat1.example.com. 

Creating a new base directory for a new instance requires the creation and copying of various directories and configuration files. 
Execute the following commands as root:

# mkdir -p /opt/tomcat-instance/tomcat1.example.com
# cd /opt/tomcat-instance/tomcat1.example.com
#
# cp -a /opt/tomcat/apache-tomcat-7.0.30/conf .
# mkdir common logs temp server shared webapps work
#
# chown -R alex:alexander /opt/tomcat-instance

Most of the remaining steps are executed as the alex user. So make sure you switch from root to alex:

# su - -s /bin/sh alex
$ id
uid=1001(alex) gid=1001(alexander) groups=1001(alexander)


Next I created an environment file for the new Tomcat instance. 
This will be useful for easily setting the environment variables when starting/stopping the new Tomcat instance:

$ vim /opt/tomcat-instance/tomcat1.env
export JAVA_HOME=/usr/java/jdk1.6.0_10
export PATH=\$JAVA_HOME/bin:\$PATH
export CATALINA_HOME=/opt/tomcat/apache-tomcat-7.0.30
export CATALINA_BASE=/opt/tomcat-instance/tomcat1.example.com


$ cat /opt/tomcat-instance/tomcat1.env
export JAVA_HOME=/usr/java/jdk1.6.0_10
export PATH=$JAVA_HOME/bin:$PATH
export CATALINA_HOME=/opt/tomcat/apache-tomcat-7.0.30
export CATALINA_BASE=/opt/tomcat-instance/tomcat1.example.com


CATALINA_HOME is the base directory of Tomcat that contains all the libraries, scripts etc. for Tomcat. This is the parent directory of the extracted Tomcat tar file.
CATALINA_BASE is the base directory of the new Tomcat instance, which in this example points to /opt/tomcat-instance/tomcat1.example.com. 

Configuring Tomcat Network Ports

Since this is the first Tomcat instance that's being created here, the default port numbers can be left unchanged in $CATALINA_BASE/conf/server.xml (/opt/tomcat-instance/tomcat1.example.com/conf/server.xml):

    <Server port="8005" shutdown="SHUTDOWN">

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

Please keep mind, these port numbers will have to be changed for the second Tomcat instance though.

Starting First Tomcat Instance

To start the newly created Tomcat instance, ensure that the environment variables are set for the new instance and execute the startup script:

$ source /opt/tomcat-instance/tomcat1.env
$ $CATALINA_HOME/bin/startup.sh
Using CATALINA_BASE:   /opt/tomcat-instance/tomcat1.example.com
Using CATALINA_HOME:   /opt/tomcat/apache-tomcat-7.0.30
Using CATALINA_TMPDIR: /opt/tomcat-instance/tomcat1.example.com/temp
Using JRE_HOME:       /usr/java/jdk1.6.0_10


If everything has been configured correctly, I should now see an empty white page when opening the URL http://localhost:8080. Note that instead of localhost you should also be able to use the name of your server. 
If you get an error in the browser instead of an empty page, check the log files under $CATALINA_BASE/logs (/opt/tomcat-instance/tomcat1.example.com/logs). 
Note that since CATALINA_BASE has been changed for the new Tomcat instance, the logs are no longer written to /opt/tomcat/apache-tomcat-7.0.30/logs. 

Setting up Web Application Layout

the first Tomcat instance was setup under the base directory $CATALINA_BASE (/opt/tomcat-instance/tomcat1.example.com). 
Now I create a new directory called "WEB1" under $CATALINA_BASE/webapps which will become the root directory for the first web application, that is $CATALINA_BASE/webapps/WEB1. 
In Tomcat web application root directories are created under $CATALINA_BASE/webapps by default.

$ mkdir $CATALINA_BASE/webapps/WEB1

Configuring Web Application

To configure Tomcat to recognize the new web application under $CATALINA_BASE/webapps/WEB1 (/opt/tomcat-instance/tomcat1.example.com/webapps/WEB1), the $CATALINA_BASE/conf/server.xml file needs to be edited. 
This is done by adding a new Context element with the path and docBase attributes. 
Note that Tomcat refers to webapps as "context". So Context here represents the configuration of a web application. 
The path attribute is the application name used within the URL, and the docBase attribute is the absolute path name of the new web application root under $CATALINA_BASE/webapps:

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">

        <Context docBase="WEB1" path="/web1"/>

In this example you can see that appBase already points to webapps by default, that is $CATALINA_BASE/webapps. 
The newly added path attribute points to the sales directory under $CATALINA_BASE/webapps which is the location for the new application. 
And the docBase attribute is set to web1 which stands for the application name within the URL, i.e. "http://localhost:8080/web1". 
Make sure to add this new Context element inside the Host container element for 'localhost' which is the default host name. 

Home Page for Web Application

To have a starting page for the new web application, you can simply create and add a index.html file under the web application's root directory $CATALINA_BASE/webapps/WEB1 (/opt/tomcat-instance/tomcat1.example.com/webapps/WEB1). 
You could also create your own JSP page here. For testing purposes here is a simple index.html example for the new application:

$ vim $CATALINA_BASE/webapps/WEB1/index.html 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<HTML>
<HEAD><META http-equiv=Content-Type content="text/html"></HEAD>
<BODY>
<H3>TOMCAT 1 - WEB1</H3>
</BODY>
</HTML>
$

Restarting First Tomcat Instance

Now check whether the new web application has been configured correctly. To do that, run the following commands to restart the new Tomcat instance:

$ source /opt/tomcat-instance/tomcat1.env
$ $CATALINA_HOME/bin/shutdown.sh
$ $CATALINA_HOME/bin/startup.sh

If everything is right, then you should now see the default home page for the new web application when opening the URL http://localhost:8080/web1. 
Instead of localhost you should also be able to use the name of your server. If you get the error 'java.net.ConnectException: Connection refused' when you shutdown Tomcat, then Tomcat was probably not running. 
If you don't see the home page, check the log files under $CATALINA_BASE/logs. 

Deploying Java Servlet for Web Application in First Tomcat Instance

Setting up Java Servlet Layout

To follow the Java Servlet Specification for the new "WEB1" web application, I created the class directory for the Java class files under the new directory $CATALINA_BASE/webapps/WEB1/WEB-INF. 
The WEB-INF directory is protected from access by browsers, meaning they are unbrowsable and safe from client views. 
The classes directory under WEB-INF is where web components and server-side utility classes should go. To create the WEB-INF and classes directories, run the following command:

$ mkdir -p $CATALINA_BASE/webapps/WEB1/WEB-INF/classes

JAR Files

Most Java servlets also need JAR (Java ARchive) files which should be put under the lib directory. Since it's a good practice to keep the application separate from the Tomcat distribution directory tree, 
I created a new lib directory under $CATALINA_BASE/webapps/WEB1/WEB-INF which is consistent with WAR's hierarchical directory structure.

$ mkdir $CATALINA_BASE/webapps/WEB1/WEB-INF/lib

The Java servlet example below requires the servlet-api.jar JAR file. This JAR is already available in the Tomcat distribution directory tree $CATALINA_HOME/lib. 
You could copy this JAR file to the application's new lib directory $CATALINA_BASE/webapps/WEB1/WEB-INF/lib, but then you would get the following warning in the $CATALINA_BASE/logs/catalina.out log file when you startup Tomcat: 

INFO: validateJarFile(/opt/tomcat-instance/tomcat1.example.com/webapps/WEB1/WEB-INF/lib/servlet-api.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class 

Tomcat shows this warning since it tries now to load the JAR file twice, first from $CATALINA_HOME/lib and then from $CATALINA_BASE/webapps/WEB1/WEB-INF/lib. 
Even though it's not going to be a problem for Tomcat, it's better not to keep JARs in two places. Since the servlet-api.jar JAR file already exists in the Tomcat distribution directory, I did not copy it to the $CATALINA_BASE/webapps/sales/WEB-INF/lib directory. I use this directory for application specific JARs that don't come with the Tomcat distribution. You could also remove the JAR in $CATALINA_HOME/lib but remember that it will reappier the next time you upgrade the Tomcat software. 

Creating a Java Servlet

Since server-side classes are supposed to go to the WEB-INF/classes directory, I created the following class file example under $CATALINA_BASE/webapps/WEB1/WEB-INF/classes (/opt/tomcat-instance/tomcat1.example.com/webapps/WEB1/WEB-INF/classes) 
and saved it as Web1Class.java:

$ cat $CATALINA_BASE/webapps/sales/WEB-INF/classes/Web1Class.java
import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class Web1Class extends HttpServlet {

    public void doGet(HttpServletRequest request, HttpServletResponse response)
    throws IOException, ServletException
    {
        response.setContentType("text/html");
        PrintWriter out = response.getWriter();
        out.println("<html>");
        out.println("<head>");
        out.println("<title>Web1 Class Page</title>");
        out.println("</head>");
        out.println("<body>");
        out.println("<h1>Executing Web1Class ...</h1>");
        out.println("</body>");
        out.println("</html>");
    }
}

To compile the new Java servlet, the servlet-api.jar JAR file is needed which can be specified with either the -classpath option or the CLASSPATH environment variable. 
The -classpath option for SDK tools is preferred over the CLASSPATH environment variable since it can be set individually for each application without affecting others. 
In the following example I specify the path of the class directory with the basename '*' (if you are unfamiliar with basename, see 'man basename'). 
This is equivalent to specifying all files with the extensions .jar or .JAR files in the directory and therefore individual JAR files like servlet-api.jar don't need to be specified. 

The following command should now compile the Java servlet without errors:

$ cd $CATALINA_BASE/webapps/WEB1/WEB-INF/classes
$ javac -classpath "$CATALINA_HOME/lib/*" Web1Class.java
$ ls
Web1Class.class  Web1Class.java

Configuring the Java Servlet

To configure servlets and other components for an application, an XML file called web.xml needs to be configured. The format of this file is defined in the Java Servlet Specification. 
In Tomcat, this file exists in two place:
  $CATALINA_BASE/conf/web.xml
  $CATALINA_BASE/webapps/{your-appname}/WEB-INF/web.xml
The first one is the default web.xml file which is the base for all web applications in a Tomcat instance, and the latter one is for the web application where WEB-INF resides for overwriting application specific settings. 

For the newly created Java servlet "Web1Class" I created a new web.xml file under $CATALINA_BASE/webapps/WEB1/WEB-INF:
$ cat $CATALINA_BASE/webapps/WEB1/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">

  <servlet>
    <servlet-name>servlet_web1class</servlet-name>
    <servlet-class>Web1Class</servlet-class>
  </servlet>

  <servlet-mapping>
    <servlet-name>servlet_web1class</servlet-name>
    <url-pattern>/execute</url-pattern>
  </servlet-mapping>

</web-app>

For each servlet there is a <servlet> element. It identifies the servlet name (<servlet-name>) and the Java class name (<servlet-class>). 
The servlet mapping (<servlet-mapping>) maps a URI to the servlet name (<servlet-name>). In the above example "/execute" in "http://localhost:8080/web1/execute" maps to "servlet_web1class" which points to the "Web1Class" servlet class. 
Note that the order of these elements is important. So when you open the URL "http://localhost:8080/web1/execute", the "Web1Class" Java servlet will be executed. 

In the following example I updated the $CATALINA_BASE/webapps/WEB1/index.html file to provide an entry point to the new Java servlet:

$ cat $CATALINA_BASE/webapps/WEB1/index.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<HTML>
<HEAD><META http-equiv=Content-Type content="text/html"></HEAD>
<BODY>
<H3>TOMCAT 1 - WEB1</H3>
<a href="/web1/execute">Execute Web1Class</a>
</BODY>
</HTML>
$

Testing and Executing the Java Servlet

Note that if you run javac with the -classpath option or the CLASSPATH environment variable in the same shell before you startup Tomcat, you will get java.lang.NoClassDefFoundError / java.lang.ClassNotFoundException errors in your browser when you execute a servlet. To avoid this, simply re-login as the alex user before you startup Tomcat:

# su - -s /bin/sh alex
$ source /opt/tomcat-instance/tomcat1.env
$ $CATALINA_HOME/bin/shutdown.sh
$ $CATALINA_HOME/bin/startup.sh

After Tomcat restarted, open the URL http://localhost:8080/web1 (or use the server name instead of localhost) and you should see the "Execute Web1Class" link. Clicking on this link should invoke the Java servlet and display "Executing Web1Class..." in your browser. 
If you are presented with an empty page instead, review the above steps and make sure you didn't miss a step. Check also the log files under $CATALINA_BASE/logs.