SSL Server cert import for Java apps

The below is to document the procedures to import Server certs into the JVM.
This is to avoid exceptions like the below one:


Exception Message: sun.security.validator.ValidatorException: PKIX path building failed

The above is specifically showing an issue with the certification path that should be included in the JVM cacerts.
For info about the JVM certificate files please check this URL :
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManager

Steps :

1- Acquire the needed server certificate from the server URL if possible by exporting it from Firefox.
Please make sure you are getting all the certificate path exported.
This would mean that you may need to export root cert, intermediate cert and the leaf server cert.
Would be better to obtain the cert from the server support team, would be much better if possible.

2-Import Certs:
 /opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -import -trustcacerts -v -alias RSAV2 -file /tmp/RSAV2.cer

The above inserts an intermediate cert signed by RSA cert. authority and importing it with the user defined alias  RSAV2.

You might need to import all the certs defined in the server cert path.

3- List Certs:
/opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -list
You should be able to see the certs that you have imported in the list along with the date the cert was imported.

 Also below is a very useful link for doing SSL cert and key debuging in case you are setting up the server and creating a certificate.

http://www.sslshopper.com/article-most-common-openssl-commands.html

Clustering issues on SQLFire and RabbitMQ

I have been seeing much clustering issues in the last months on both RabbitMQ and SQLFire and both are Pivotal products which are opensource.
Seems like both products have issues with network latency that would cause Split brain issues in the cluster and could lead to potential data loss.

In order to be able to tell when such issues happen i have used the following approches to be able to tell if we have a cluster issue:

1- Integrate Hypric monitoring with SQLFire & RabbitMQ components.
2- For SQLFire, we can make use of the following system query:

 cat get_members.sql
select ID,KIND from sys.members order by KIND;

Running this query with commandline :
{HOME}/sf/sqlf run -client-bind-address=${HOSTNAME} -client-port=1527 -user=myapp -password=myapp -file=get_members.sql

Parsing this output would allow knowing the current number of cluster members.
if any split happens the output of this query will be differant.

3- For RabbitMQ, used a more radical way to do the monitoring.
RabbitMQ nodes will be always talking to each other, so the warning is based on the number of connections that each node has towords the sister node in the cluster:

    CON_COUNT=`ssh -q rmquser@rmqnode01 netstat -p 2>/dev/null|grep -i est |tr -s " "|cut -d" " -f5,7|grep rmqnode|cut -d"." -f1,4 --output-delimiter=" "|cut -d" " -f1,3 |sort |uniq -c|wc -l`

This will get the number of connections from rmqnode01 to all other cluster members.
The count should be number of clustermembers - 1

If the number is less, then we have a split brain issue.
Also  RabbitMQ management console tell you at once that there is an issue.

A future thing is to capture the warning from the RabbitMQ managment console directory.

Splitting a catalina log on Tread dumps

This is a useful script for extracting and splitting thread dumps out of the catalina.out or any other output file.
Java thread dump stack trance is generated by sending hup signal to the jvm using kill -3.
This will generate tread dump on the console log.

The below script collects the dumps in individual files facilitating investigation.
Below is tested on Java 1.6.

cat split_THD.sh
set -x
#
#
#
#
#


FILE=$1

STARTLINE=(`grep -n "Full thread dump Java HotSpot(TM)" ${FILE} |cut -d":" -f1`)
ENDLINE=(`grep -n "JNI global references" ${FILE} |cut -d":" -f1`)

for ((i=0;i<${#STARTLINE[@]};i++))
do
        tail -n +${STARTLINE[$i]} ${FILE} | head -n $((${ENDLINE[$i]} - ${STARTLINE[$i]} + 10)) >thd_${i}
done

Using Piping in a script

Just to document it, since i forget those things easily.
Below is an example email script based on sendmail.
The cat command will read anything coming from a pipe to the MESSAGE variable.

mail.sh:

# send am email to alert if something is not working
MESSAGE=`cat -`

/usr/sbin/sendmail.sendmail -i -t << ENDL
From: "Alert" <admin@khofo02.beren.tst>
To: <sherif.abdelfattah@beren.tst>
Subject:${1}

${MESSAGE}
ENDL

Monitoring RabbitMQ message Queues with NodeJS

RabbitMQ exposed a JSON based API from a web interface.
All the useful output is provided as Json documents that can be parsed by any application for monitoring purposed.
Best as easiest approach to do command line monitoring for RabbitMQ is to used NodeJS to encode the JSON output.
below is a quick sample:

q.js:

var fs = require('fs');
var file = process.argv[2];

fs.readFile(file, 'utf8', function (err, data)
{
if (err)
{
console.log('Error: ' + err);
return;
}

data = JSON.parse(data);

console.log(data[0].name,data[4].messages_ready);
console.log(data[1].name,data[4].messages_ready);
console.log(data[2].name,data[4].messages_ready);
console.log(data[3].name,data[4].messages_ready);

});


rmqmon.sh:

function tableit()
{
    echo "Please check Prod RabbitMQ. Queue counts are none Zero."
    echo " "
    printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"
        printf "|%-25s|%-5s| \r\n" "QueueName" "Count"
        printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"
        for i in `cat ${1}|tr " " ":"`
        do
                QUE=`echo ${i} |cut -d":" -f1`
                COUNT=`echo ${i} |cut -d":" -f2`
                #echo ${QUE}
                printf "|%-25s|%5d| \r\n" ${QUE} ${COUNT}
                #read
        done
        printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"


}




OUT_PATH=/sherif/rmqmon
NODE_PATH=/sherif/node/bin



curl  -u qmon:qmon  http://rmqnode01:15672/api/queues/ >${OUT_PATH}/q.json 2>/dev/null
${NODE_PATH}/node ${NODE_PATH}/q.js ${OUT_PATH}/q.json >${OUT_PATH}/q.status
#cat ${OUT_PATH}/q.status
HAS_NONE_ZERO=`grep -v " 0" q.status`

if [ ! -z "${HAS_NONE_ZERO}" ]
#if [ -z "${HAS_NONE_ZERO}" ]
then
    tableit ${OUT_PATH}/q.status |/sherif/mail_alert2.sh "RabbitMQ Alert"
fi
The above will provide output only if  any of the queues has messages in them.
More sophisticated NodeJS or shell scripts can be built around these.
The above tabelit function is added for more cool email look :)
using poweful formatting of C like printf shell function.
Also the mail_alert2.sh uses shell piping as input, details on this found on another post.

Exploring DotCMS Cont.

Looks like a very Robust CMS system that allows all kinds of content.
Also integrates many canvases for editing content either by typing HTML or using WUSIWUG.
The tool is very polished and the test site that comes with it is a very cool static site that is enterprise grade.

Though i didn't test how would it be able to handle lots of hits for content queries and didn't do comprehensive site creation with it.

Will continue to post more about it . . .

Exploring DotCMS

The dotcms tool is proving to be a great content management system for free.
It has all the bells and whistles out of the box.
just finished its installation and now started playing with it.
Proves to be rather solid and very well organized.

Will post more as it goes.
http://dotcms.com/