Project Cron: Example of cron job scripting in bash (Bourne) under Linux flavors Debian and Ubuntu

Purpose
The purpose of this tutorial is to illustrate a custom system cron script that contains many common operational tasks that many cron bash scripts will contain. By seeing how different operations work together under bash, you should be able to pick up cron bash scripting pretty quick instead of spending countless hours searching and piecing parts together.

A Few Helpful Tips:
Although scripting cron jobs with bash (Bourne) is essentially creating your own .sh scripts, cron scripting varies a bit:

System cron scripts will be found in /etc. You can modify the system /etc/crontab file (such as sudoedit /etc/crontab) to point to your custom system cron script located in /etc/cron.d with the schedule you want to call it with - usually the only time you want to do this is when your system cron script will not run on a hourly, daily, weekly or monthly schedule.
If your custom system cron script will run on system schedule (hourly, daily, weekly, monthly), just place your cron script in the appropriate /etc/cron.* folder (such as /etc/cron.hourly); it should be automatically detected.
Don't include the file extension such as .sh of your custom system cron script. For example, the custom system cron script included on this page is just called "flushjava".
If you need to test your cron script, you can place it in /etc/cron.d. Then, to call it, at the prompt:
root@server:~# run-parts /etc/cron.d
Typically system cron scripts will have chmod a+x permissions applied to them and the owner will be chown root.
Need to keep an .sh script you call from the cron script running while the cron script performs other processing? This script example shows how it is done with minimal overhead.

CUSTOM CRON SCRIPT:

#!/bin/bash

#title		:flushjava
#description	:The purpose of this cron job is to flush java running on the server by killing what currently exists and then starting up a new set
#description	:  of java processes.
#author		:Joe
#date		:20120327
#version	:1.0
#usage		:/etc/cron.weekly
#notes		:This script is intended to be run by the system by dropping into cron.weekly (or cron.hourly, cron.daily, cron.monthly).
#notes		:  Alternatively you can place into /etc/cron.d and run via SSH with run-parts /etc/cron.d for immediate running/testing/modification.
#notes		:  [OTHER] Get bash_version via "echo $BASH_VERSION" and bash_full via "bash --version"
#bash_version	:4.1.5(1)-release
#bash_full	:GNU bash, version 4.1.5(1)-release (x86_64-pc-linux-gnu)

# Define a few global variables
userAcct="skippy"		# User account where Hazelcast start.sh is located, and the user account to run the processes under
cronSchedule="cron.weekly"	# Type of system cron schedule (displayed in /var/log/flushjava.log)
showInteractive=0		# Useful if you are running this yourself such as with root@server:~# run-parts /etc/cron.d. 0 = show no messages
				# to screen, 1 = show messages to screen.
logProgress=1			# Useful if you want to log the steps this cron job completed to /var/log/flushjava.log.
detectRunningJavaServer=-1
detectRunningJavaHazel=-1
cDate=`date`
exCode=0
currentMsg=""
progressMsgs=""

# Discover Java Hazelcast process running on server
currentMsg="Detecting process /usr/bin/java -cp."
if [ $showInteractive = 1 ]; then echo $currentMsg; fi
if [ $logProgress = 1 ]; then progressMsgs=$currentMsg; fi
output1=`ps -C java f | grep "/usr/bin/java -cp"`
set -- $output1
# Grab the process ID: output should be PID($1) TTY($2) STAT($3) TIME($4) COMMAND($5)
pid1=$1
# Only flush if Java Hazelcast is running on server
if [ "$pid1" != "" ]; then
	currentMsg="Process found with PID("$pid1"), will attempt friendly termination."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	# Kill the process (-SIGTERM)
	kill -15 $pid1
	# Wait for a bit
	sleep 15
	# See if it is still present; should not be if the application was listening for a termination signal
	currentMsg="Detecting if process is still active."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	look1=`ps -C java f | grep "/usr/bin/java -cp"`
	set -- $look1
	pidHere1=$1
	if [ "$pidHere1" = "$pid1" ]; then
		currentMsg="Process with PID("$pidHere1") still active, will attempt forced termination."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		# Kill the process (-SIGKILL) that is still present (send output to oblivion)
		kill -9 $pidHere1 >/dev/null 2>&1
		# Wait for a bit
		sleep 15
		# See if it is still present (-f to get the PPID) but a zombie process to flush from system RAM
		currentMsg="Detecting if zombie/artifact remains."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		zombie1=`ps -C java -f | grep "<defunct>"`
		set -- $zombie1
		# Grab the parent process ID: output should be UID($1) PID($2) PPID($3) C($4) STIME($5) TTY($6) TIME($7) COMMAND($8)
		parentPid1=$3
		# Kill the parent process
		if [ "$2" = "$pid1" ]; then
			currentMsg="Zombie/artifact remains PPID("$parentPid1"), will force termination."
			if [ $showInteractive = 1 ]; then echo $currentMsg; fi
			if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
			kill -9 $parentPid1 >/dev/null 2>&1
			# Wait for a bit
			sleep 15
		fi
	else
		# Make sure a artifact/zombie process was not left behind
		currentMsg="Detecting if zombie/artifact remains."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		artifact1=`ps -C java -f | grep "<defunct>"`
		set -- $artifact1
		# Grab the parent process ID
		artifactPid1=$3
		# Kill the parent process
		if [ "$2" = "$pid1" ]; then
			currentMsg="Zombie/artifact remains PPID("$artifactPid1"), will force termination."
			if [ $showInteractive = 1 ]; then echo $currentMsg; fi
			if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
			kill -9 $artifactPid1 >/dev/null 2>&1
			# Wait for a bit
			sleep 15
		fi
	fi
fi

# Discover the Java server process running on the server
currentMsg="Detecting process java -server."
if [ $showInteractive = 1 ]; then echo $currentMsg; fi
if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
output2=`ps -C java f | grep "java -server"`
set -- $output2
# Grab the process ID
pid2=$1
# Only flush if Java is running on server
if [ "$pid2" != "" ]; then
	currentMsg="Process found with PID("$pid2"), will attempt friendly termination."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	# Kill the process
	kill -15 $pid2
	# Wait for a bit
	sleep 15
	# See if it is still present
	currentMsg="Detecting if process is still active."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	look2=`ps -C java f | grep "java -server"`
	set -- $look2
	pidHere2=$1
	if [ "$pidHere2" = "$pid2" ]; then
		currentMsg="Process with PID("$pidHere2") still active, will attempt forced termination."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		# Kill the process that is still present
		kill -9 $pidHere2 >/dev/null 2>&1
		# Wait for a bit
		sleep 15
		# See if it is still present but a zombie process to flush from system RAM
		currentMsg="Detecting if zombie/artifact remains."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		zombie2=`ps -C java -f | grep "<defunct>"`
		set -- $zombie2
		# Grab the parent process ID
		parentPid2=$3
		# Kill the parent process
		if [ "$2" = "$pid2" ]; then
			currentMsg="Zombie/artifact remains PPID("$parentPid2"), will force termination."
			if [ $showInteractive = 1 ]; then echo $currentMsg; fi
			if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
			kill -9 $parentPid2 >/dev/null 2>&1
			# Wait for a bit
			sleep 15
		fi
	else
		# Make sure a artifact/zombie process was not left behind
		currentMsg="Detecting if zombie/artifact remains."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		artifact2=`ps -C java -f | grep "<defunct>"`
		set -- $artifact2
		# Grab the parent process ID
		artifactPid2=$3
		# Kill the parent process
		if [ "$2" = "$pid2" ]; then
			currentMsg="Zombie/artifact remains PPID("$artifactPid2"), will force termination."
			if [ $showInteractive = 1 ]; then echo $currentMsg; fi
			if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
			kill -9 $artifactPid2 >/dev/null 2>&1
			# Wait for a bit
			sleep 15
		fi
	fi
fi

# Start up the "java -server" under the userAcct account
# NOTE 1: server.sh spends a lot of time scanning and setting up connections.  In order to allow this cron script to continue processing
# (since server.sh does not exit) need to wait for a while and then send a keyboard interrupt signal (2) to return control to this cron script.
# NOTE 2: Interrupts include:
# SIGHUP 1 - Hangup - allows process to continue running but will exit out of this cron script
# SIGINT 2 - Keyboard Interrupt - allows process to continue running as well as this cron script
# SIGTERM 15 - Termination Signal - request that the process terminate itself
# SIGSTOP 17, 19, 23 - stop but don't terminate the process; akin to pause
# SIGKILL 9 - have the kernel terminate the process which could result in data loss and so on but guarantees it goes away
# NOTE 3: Only works if running in the background (&)
currentMsg="Starting process java -server."
if [ $showInteractive = 1 ]; then echo $currentMsg; fi
if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
cd "/home/"$userAcct"/hazelcast/hazelcast-2.3.1/bin"
su "$userAcct" ./server.sh& sleep 30;kill -2 $!

# Make sure the "java -server" process is running
currentMsg="Detecting process java -server."
if [ $showInteractive = 1 ]; then echo $currentMsg; fi
if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
output3=`ps -C java f | grep "java -server"`
set -- $output3
pid3=$1
if [ "$pid3" = "" ]; then
	currentMsg="Process java -server failed to start."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	detectRunningJavaServer=0
else
	currentMsg="Process java -server with PID("$pid3") successfully started."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	detectRunningJavaServer=1
fi

# Start up the "/usr/bin/java -cp" under the userAcct account only if the "java -server" process is running
# NOTE 1: Also run in the background (&) to ensure this script does not get called before the prior one has started running
# NOTE 2: While this script does not "hang" like the prior one, we still need to return control to this cron script after this script runs
if [ $detectRunningJavaServer = 1 ]; then
	currentMsg="Starting process /usr/bin/java -cp."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	cd "/home/"$userAcct
	su "$userAcct" ./start.sh& sleep 5;

	# Make sure the "/usr/bin/java -cp" process is running
	currentMsg="Detecting process /usr/bin/java -cp."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	output4=`ps -C java f | grep "/usr/bin/java -cp"`
	set -- $output4
	pid4=$1
	if [ "$pid4" = "" ]; then
		currentMsg="Process /usr/bin/java -cp failed to start."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		detectRunningJavaHazel=0
	else
		currentMsg="Process /usr/bin/java -cp with PID("$pid4") successfully started."
		if [ $showInteractive = 1 ]; then echo $currentMsg; fi
		if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
		detectRunningJavaHazel=1
	fi
fi

# If the processes are not running, log the problem, email, etcetera
if [ $detectRunningJavaServer != 1 ] || [ $detectRunningJavaHazel != 1 ]; then
	currentMsg="A process did not start, logging failure."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	exCode=1
	# NOTE 1: Common operators like  . + not allowed when joining string fragments together
	strCompile=$cDate" ::: One or more processes could not be started by the cron job /etc/"$cronSchedule"/flushjava:  "
	if [ $detectRunningJavaServer != 1 ]; then
		strCompile=$strCompile"java -server could not be started ("$detectRunningJavaServer").  "
	fi
	if [ $detectRunningJavaHazel != 1 ]; then
		strCompile=$strCompile"/usr/bin/java -cp could not be started ("$detectRunningJavaHazel")."
	fi
	# Save to log file
	currentMsg="Saving failure to log."
	if [ $showInteractive = 1 ]; then echo $currentMsg; fi
	if [ $logProgress = 1 ]; then progressMsgs=$progressMsgs" "$currentMsg; fi
	logFile="/var/log/flushjava.log"
	if [ -f $logFile ]; then
		# Append data to file
		echo $strCompile >> $logFile
	else
		# Create new file
		touch $logFile
		# Assign root as owner
		chown root $logFile
		# Assign 0644 permissions to the file
		chmod u+rw,g+r,o+r $logFile
		# Dump content into file
		echo $strCompile > $logFile
	fi
fi

# Save cron progress if specified
if [ $logProgress = 1 ]; then
	progressMsgs=$cDate" [CRON PROGRESS DUMP] "$progressMsgs
	logFile="/var/log/flushjava.log"
	if [ -f $logFile ]; then
		# Append data to file
		echo $progressMsgs >> $logFile
	else
		# Create new file
		touch $logFile
		# Assign root as owner
		chown root $logFile
		# Assign 0644 permissions to the file
		chmod u+rw,g+r,o+r $logFile
		# Dump content into file
		echo $progressMsgs > $logFile
	fi
fi

# Exit
exit $exCode