Splitting a catalina log on Tread dumps

This is a useful script for extracting and splitting thread dumps out of the catalina.out or any other output file.
Java thread dump stack trance is generated by sending hup signal to the jvm using kill -3.
This will generate tread dump on the console log.

The below script collects the dumps in individual files facilitating investigation.
Below is tested on Java 1.6.

cat split_THD.sh
set -x
#
#
#
#
#


FILE=$1

STARTLINE=(`grep -n "Full thread dump Java HotSpot(TM)" ${FILE} |cut -d":" -f1`)
ENDLINE=(`grep -n "JNI global references" ${FILE} |cut -d":" -f1`)

for ((i=0;i<${#STARTLINE[@]};i++))
do
        tail -n +${STARTLINE[$i]} ${FILE} | head -n $((${ENDLINE[$i]} - ${STARTLINE[$i]} + 10)) >thd_${i}
done

Using Piping in a script

Just to document it, since i forget those things easily.
Below is an example email script based on sendmail.
The cat command will read anything coming from a pipe to the MESSAGE variable.

mail.sh:

# send am email to alert if something is not working
MESSAGE=`cat -`

/usr/sbin/sendmail.sendmail -i -t << ENDL
From: "Alert" <admin@khofo02.beren.tst>
To: <sherif.abdelfattah@beren.tst>
Subject:${1}

${MESSAGE}
ENDL

Monitoring RabbitMQ message Queues with NodeJS

RabbitMQ exposed a JSON based API from a web interface.
All the useful output is provided as Json documents that can be parsed by any application for monitoring purposed.
Best as easiest approach to do command line monitoring for RabbitMQ is to used NodeJS to encode the JSON output.
below is a quick sample:

q.js:

var fs = require('fs');
var file = process.argv[2];

fs.readFile(file, 'utf8', function (err, data)
{
if (err)
{
console.log('Error: ' + err);
return;
}

data = JSON.parse(data);

console.log(data[0].name,data[4].messages_ready);
console.log(data[1].name,data[4].messages_ready);
console.log(data[2].name,data[4].messages_ready);
console.log(data[3].name,data[4].messages_ready);

});


rmqmon.sh:

function tableit()
{
    echo "Please check Prod RabbitMQ. Queue counts are none Zero."
    echo " "
    printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"
        printf "|%-25s|%-5s| \r\n" "QueueName" "Count"
        printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"
        for i in `cat ${1}|tr " " ":"`
        do
                QUE=`echo ${i} |cut -d":" -f1`
                COUNT=`echo ${i} |cut -d":" -f2`
                #echo ${QUE}
                printf "|%-25s|%5d| \r\n" ${QUE} ${COUNT}
                #read
        done
        printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"


}




OUT_PATH=/sherif/rmqmon
NODE_PATH=/sherif/node/bin



curl  -u qmon:qmon  http://rmqnode01:15672/api/queues/ >${OUT_PATH}/q.json 2>/dev/null
${NODE_PATH}/node ${NODE_PATH}/q.js ${OUT_PATH}/q.json >${OUT_PATH}/q.status
#cat ${OUT_PATH}/q.status
HAS_NONE_ZERO=`grep -v " 0" q.status`

if [ ! -z "${HAS_NONE_ZERO}" ]
#if [ -z "${HAS_NONE_ZERO}" ]
then
    tableit ${OUT_PATH}/q.status |/sherif/mail_alert2.sh "RabbitMQ Alert"
fi
The above will provide output only if  any of the queues has messages in them.
More sophisticated NodeJS or shell scripts can be built around these.
The above tabelit function is added for more cool email look :)
using poweful formatting of C like printf shell function.
Also the mail_alert2.sh uses shell piping as input, details on this found on another post.