Friday, June 7, 2013

Application log monitoring and Oracle particular Database monitoring thru bash scripts

Day before yesterday, i have been assigned one task to write a script which will do below..
Task
1. Monitor Log say, java_abc.log of the java application
2. If error reported in java_abc.log regarding db connection
3. check db connection remotely using sqlplus login command in three intervals 5 sec,10 sec, 15 sec
4. if db connection is restored then restart the java app and notify thru email to respective teams
5. If db connection not restored then simply notify dba and other respective teams.

Required :
Password free ssh between DBSERVER and APPSERVER.

                                                Password less ssh
DBSERVER <------------------------------------------------------> APPSERVER

Scripts On APPSERVER(click on script name to open)
check_log (nagios plugin script, can be found at nagios site)
java_log_monitor
java_app_restart <--- your application restart script

Script On DBSERVER(click on script name to open)
dbMONITOR

Flow
1. APPSERVER Crontab will execute java_log_monitor every 5 mins
2. java_log_monitor will call check_log script to check java_abc.log to search below error pattern
ORA (to check all oracle errors like ORA-12528, TNS:listener: all appropriate instances are blocking new connections) and
java.lang.RuntimeException: Couldn't get the connection
3. If above patterns reported in java_abc.log file, it will execute the dbMonitor script on DBSERVER
4. dbMonitor will return 0 (connection ok) or 1 (connection not ok) to java_log_monitor script
5. If dbMonitor returns 0 (connection ok), then java_abc_monitor will call java_app_restart script to restart java application and will notify respective teams thru email.
6. If dbMonitor returns 1 (connection not ok), then java_log_monitor will again call dbMonitor on DBSERVER three times (5 sec, 10 sec, 15 sec) to check db status.
7. if dbMonitor still returns 1 (connection not ok) then java_log_monitor will notify respective teams.

Hope this info will help someone...
Please email me if you have any query....

No comments:

Post a Comment