SSL Server cert import for Java apps

The below is to document the procedures to import Server certs into the JVM.
This is to avoid exceptions like the below one:


Exception Message: sun.security.validator.ValidatorException: PKIX path building failed

The above is specifically showing an issue with the certification path that should be included in the JVM cacerts.
For info about the JVM certificate files please check this URL :
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManager

Steps :

1- Acquire the needed server certificate from the server URL if possible by exporting it from Firefox.
Please make sure you are getting all the certificate path exported.
This would mean that you may need to export root cert, intermediate cert and the leaf server cert.
Would be better to obtain the cert from the server support team, would be much better if possible.

2-Import Certs:
 /opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -import -trustcacerts -v -alias RSAV2 -file /tmp/RSAV2.cer

The above inserts an intermediate cert signed by RSA cert. authority and importing it with the user defined alias  RSAV2.

You might need to import all the certs defined in the server cert path.

3- List Certs:
/opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -list
You should be able to see the certs that you have imported in the list along with the date the cert was imported.

 Also below is a very useful link for doing SSL cert and key debuging in case you are setting up the server and creating a certificate.

http://www.sslshopper.com/article-most-common-openssl-commands.html

Clustering issues on SQLFire and RabbitMQ

I have been seeing much clustering issues in the last months on both RabbitMQ and SQLFire and both are Pivotal products which are opensource.
Seems like both products have issues with network latency that would cause Split brain issues in the cluster and could lead to potential data loss.

In order to be able to tell when such issues happen i have used the following approches to be able to tell if we have a cluster issue:

1- Integrate Hypric monitoring with SQLFire & RabbitMQ components.
2- For SQLFire, we can make use of the following system query:

 cat get_members.sql
select ID,KIND from sys.members order by KIND;

Running this query with commandline :
{HOME}/sf/sqlf run -client-bind-address=${HOSTNAME} -client-port=1527 -user=myapp -password=myapp -file=get_members.sql

Parsing this output would allow knowing the current number of cluster members.
if any split happens the output of this query will be differant.

3- For RabbitMQ, used a more radical way to do the monitoring.
RabbitMQ nodes will be always talking to each other, so the warning is based on the number of connections that each node has towords the sister node in the cluster:

    CON_COUNT=`ssh -q rmquser@rmqnode01 netstat -p 2>/dev/null|grep -i est |tr -s " "|cut -d" " -f5,7|grep rmqnode|cut -d"." -f1,4 --output-delimiter=" "|cut -d" " -f1,3 |sort |uniq -c|wc -l`

This will get the number of connections from rmqnode01 to all other cluster members.
The count should be number of clustermembers - 1

If the number is less, then we have a split brain issue.
Also  RabbitMQ management console tell you at once that there is an issue.

A future thing is to capture the warning from the RabbitMQ managment console directory.