Converting variable base DS info into JNDI

This was quite an issue for me since a while.
There are 3 ways to put in the Data source info into a tomcat.
1- put it in context.xml directly which will save time and effort.
2- put into server.xml as global naming resources and then add an entry into context.xml for global resource link
3- put as variables into catalina.properties and then link them in server.xml and context.xml which is the longest way !

below is a script that changes the variable notation to the server.xml notation.
Also will print out the resource link for context.xml:

below is how the ds info got from catalina.properties look like:

[root@Beren DS]# head ds_info
aaa.conf.datasource1.name=configDataSource
aaa.conf.datasource1.url=jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = orcl_host)(PORT = 1527))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = orcl)))
aaa.conf.datasource1.username=sherif
aaa.conf.datasource1.password=mydspassword

aaa.conf.datasource2.name=tstDataSource
aaa.conf.datasource2.url=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=orcl_host)(PORT=1700))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=test)))
aaa.conf.datasource2.username=test
aaa.conf.datasource2.password=test
[root@Beren DS]#

Below is the script:

[root@Beren DS]# cat fix.sh                             
#set -x                                                 
OLD_IFS=${IFS}                                          
IFS='                                                   
'                                                       
for line in `cat ds_info`                               
do                                                      
        VAR=`echo ${line}|cut -d"=" -f1|cut -d"." -f4`  
        VAL="`echo ${line}|cut -d"=" -f2-`"             

        if [ "${VAR}" = "name" ]
        then                   
                echo '<Resource name="jdbc/'"${VAL}"'"'
                echo 'auth="Container" type="javax.sql.DataSource"'
        fi                                                        
        if [ "${VAR}" = "url" ]                                   
        then
                echo 'url="'"${VAL}"'"'
                echo 'driverClassName="oracle.jdbc.driver.OracleDriver"'
                echo 'maxActive="80"'
                echo 'maxIdle="20"'
                echo 'maxWait="10000"'
        fi
        if [ "${VAR}" = "username" ]
        then
                echo 'username="'"${VAL}"'"'
        fi
        if [ "${VAR}" = "password" ]
        then
                echo 'password="'""${VAL}'"'
                echo 'removeAbandoned="true"'
                echo 'removeAbandonedTimeout="60"'
                echo 'logAbandoned="true"'
                echo 'validationQuery="SELECT SYSDATE FROM DUAL"/>'
                echo " "

        fi
        #read
done

echo ""
echo "Context.xml"
echo ""

for line in `cat ds_info|grep "emc.conf.datasource.*\.name"`
do
        VAR=`echo ${line}|cut -d"=" -f1|cut -d"." -f4`
        VAL="`echo ${line}|cut -d"=" -f2-`"

        if [ "${VAR}" = "name" ]
        then
                echo ' <ResourceLink name="jdbc/'${VAL}'" global="jdbc/'${VAL}'" type="javax.sql.DataSource"/>'
        fi
        #read
done
[root@Beren DS]#

Changing JVM options using a shell script

The below script is used to customize a TC sever setenv.sh for multiple TC instances.
The script reads a config file and applys this to the setenv and pushes it to the instace loaction.


[sherif@khofo05 ~ ]$ cat fixSetenv.sh
OLD_IFS=${IFS}
IFS='
'
for INST in `cat tc_nodes |grep -v "^#"`
do 
    USER=`echo ${INST} |cut -d":" -f1`
    TC_INST=`echo ${INST} |cut -d":" -f2|cut -d"@" -f1`
    HOST=`echo ${INST} |cut -d":" -f2| cut -d "_" -f1`
    HTTP_PORT=`echo ${INST} |cut -d "_" -f2|cut -d"@" -f1`
    HOME_DIR=`echo ${INST} |cut -d"@" -f2`
    JVM_PAR=`echo ${INST} |cut -d"@" -f3`
    NEW_JAVA_HOME="JAVA_HOME=${HOME_DIR}/springsource/jdk1.7.0_21"

     OLD_JVM_PAR=`grep "JVM_OPTS=" setenv.sh|tr -d '\n'`
    OLD_JAVA_HOME=`grep "JAVA_HOME=" setenv.sh|tr -d '\n'`
IFS=${OLD_IFS}
#echo "---------------"
echo "working on ${HOST}"
echo "sed -- 's%${OLD_JVM_PAR}%${JVM_PAR}%g' setenv.sh |" >sed_cmd
echo "sed -- 's%${OLD_JAVA_HOME}%${NEW_JAVA_HOME}%g'" >>sed_cmd
echo "creating custome setenv "
bash ./sed_cmd >setenv.sh_${TC_INST}
scp setenv.sh_${TC_INST}  ${USER}@${HOST}:${HOME_DIR}/springsource/vfabric-tc-server-standard-2.9.2.RELEASE/${TC_INST}/bin/setenv.sh
echo "sent to ${TC_INST}
Done"
done
IFS=${OLD_IFS}
[sherif@khofo05 ~ ]$

The instances config file is named tc_nodes.
This file looks like this:

username1:hostname01_8080@/home/dir/of/username1@JVM_OPTS="-server -Xms4096m -Xmx4096m -Xss192k -XX:MaxPermSize=512m -XX:PermSize=256m -XX:ParallelGCThreads=2 -XX:NewSize=256m -XX:MaxNewSize=256m -Xnoclassgc -XX:SurvivorRatio=14 -Xloggc:${HOME}/logs/gc.log"

This is parsed for the parameters used by the script.
I found it more difficult to run the sed commads from within the same script. so to solve this the fixsetenv.sh creates another script and runs it.
this is the sed_cmd script.
this one looks like this:

[sherif@khofo05 ~]$ cat sed_cmd
sed -- 's%JVM_OPTS="-Xmx512M -Xss256K"%JVM_OPTS="-server -Xms4096m -Xmx4096m -Xss192k -XX:MaxPermSize=512m -XX:PermSize=256m -XX:ParallelGCThreads=2 -XX:NewSize=256m -XX:MaxNewSize=256m -Xnoclassgc -XX:SurvivorRatio=14 -Xloggc:${HOME}/logs/gc.log"%g' setenv.sh |
sed -- 's%JAVA_HOME="/homedire1/springsource/jdk1.7.0_21"%JAVA_HOME=/homedire1/springsource/jdk1.7.0_21%g'
[sherif@khofo05 ~]$

The above made it possible to do custom changes based on the config files.
In order to come around parsing the whole JVM_OPTs with the spaces and things like that I had to change the Internal field separator.
This made it possible to accommodate the spaces.


Automation is fun :)

SSL Server cert import for Java apps

The below is to document the procedures to import Server certs into the JVM.
This is to avoid exceptions like the below one:


Exception Message: sun.security.validator.ValidatorException: PKIX path building failed

The above is specifically showing an issue with the certification path that should be included in the JVM cacerts.
For info about the JVM certificate files please check this URL :
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManager

Steps :

1- Acquire the needed server certificate from the server URL if possible by exporting it from Firefox.
Please make sure you are getting all the certificate path exported.
This would mean that you may need to export root cert, intermediate cert and the leaf server cert.
Would be better to obtain the cert from the server support team, would be much better if possible.

2-Import Certs:
 /opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -import -trustcacerts -v -alias RSAV2 -file /tmp/RSAV2.cer

The above inserts an intermediate cert signed by RSA cert. authority and importing it with the user defined alias  RSAV2.

You might need to import all the certs defined in the server cert path.

3- List Certs:
/opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -list
You should be able to see the certs that you have imported in the list along with the date the cert was imported.

 Also below is a very useful link for doing SSL cert and key debuging in case you are setting up the server and creating a certificate.

http://www.sslshopper.com/article-most-common-openssl-commands.html

Clustering issues on SQLFire and RabbitMQ

I have been seeing much clustering issues in the last months on both RabbitMQ and SQLFire and both are Pivotal products which are opensource.
Seems like both products have issues with network latency that would cause Split brain issues in the cluster and could lead to potential data loss.

In order to be able to tell when such issues happen i have used the following approches to be able to tell if we have a cluster issue:

1- Integrate Hypric monitoring with SQLFire & RabbitMQ components.
2- For SQLFire, we can make use of the following system query:

 cat get_members.sql
select ID,KIND from sys.members order by KIND;

Running this query with commandline :
{HOME}/sf/sqlf run -client-bind-address=${HOSTNAME} -client-port=1527 -user=myapp -password=myapp -file=get_members.sql

Parsing this output would allow knowing the current number of cluster members.
if any split happens the output of this query will be differant.

3- For RabbitMQ, used a more radical way to do the monitoring.
RabbitMQ nodes will be always talking to each other, so the warning is based on the number of connections that each node has towords the sister node in the cluster:

    CON_COUNT=`ssh -q rmquser@rmqnode01 netstat -p 2>/dev/null|grep -i est |tr -s " "|cut -d" " -f5,7|grep rmqnode|cut -d"." -f1,4 --output-delimiter=" "|cut -d" " -f1,3 |sort |uniq -c|wc -l`

This will get the number of connections from rmqnode01 to all other cluster members.
The count should be number of clustermembers - 1

If the number is less, then we have a split brain issue.
Also  RabbitMQ management console tell you at once that there is an issue.

A future thing is to capture the warning from the RabbitMQ managment console directory.

Splitting a catalina log on Tread dumps

This is a useful script for extracting and splitting thread dumps out of the catalina.out or any other output file.
Java thread dump stack trance is generated by sending hup signal to the jvm using kill -3.
This will generate tread dump on the console log.

The below script collects the dumps in individual files facilitating investigation.
Below is tested on Java 1.6.

cat split_THD.sh
set -x
#
#
#
#
#


FILE=$1

STARTLINE=(`grep -n "Full thread dump Java HotSpot(TM)" ${FILE} |cut -d":" -f1`)
ENDLINE=(`grep -n "JNI global references" ${FILE} |cut -d":" -f1`)

for ((i=0;i<${#STARTLINE[@]};i++))
do
        tail -n +${STARTLINE[$i]} ${FILE} | head -n $((${ENDLINE[$i]} - ${STARTLINE[$i]} + 10)) >thd_${i}
done

Using Piping in a script

Just to document it, since i forget those things easily.
Below is an example email script based on sendmail.
The cat command will read anything coming from a pipe to the MESSAGE variable.

mail.sh:

# send am email to alert if something is not working
MESSAGE=`cat -`

/usr/sbin/sendmail.sendmail -i -t << ENDL
From: "Alert" <admin@khofo02.beren.tst>
To: <sherif.abdelfattah@beren.tst>
Subject:${1}

${MESSAGE}
ENDL

Monitoring RabbitMQ message Queues with NodeJS

RabbitMQ exposed a JSON based API from a web interface.
All the useful output is provided as Json documents that can be parsed by any application for monitoring purposed.
Best as easiest approach to do command line monitoring for RabbitMQ is to used NodeJS to encode the JSON output.
below is a quick sample:

q.js:

var fs = require('fs');
var file = process.argv[2];

fs.readFile(file, 'utf8', function (err, data)
{
if (err)
{
console.log('Error: ' + err);
return;
}

data = JSON.parse(data);

console.log(data[0].name,data[4].messages_ready);
console.log(data[1].name,data[4].messages_ready);
console.log(data[2].name,data[4].messages_ready);
console.log(data[3].name,data[4].messages_ready);

});


rmqmon.sh:

function tableit()
{
    echo "Please check Prod RabbitMQ. Queue counts are none Zero."
    echo " "
    printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"
        printf "|%-25s|%-5s| \r\n" "QueueName" "Count"
        printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"
        for i in `cat ${1}|tr " " ":"`
        do
                QUE=`echo ${i} |cut -d":" -f1`
                COUNT=`echo ${i} |cut -d":" -f2`
                #echo ${QUE}
                printf "|%-25s|%5d| \r\n" ${QUE} ${COUNT}
                #read
        done
        printf "|%-25s|%-5s| \r\n" "-------------------------" "-----"


}




OUT_PATH=/sherif/rmqmon
NODE_PATH=/sherif/node/bin



curl  -u qmon:qmon  http://rmqnode01:15672/api/queues/ >${OUT_PATH}/q.json 2>/dev/null
${NODE_PATH}/node ${NODE_PATH}/q.js ${OUT_PATH}/q.json >${OUT_PATH}/q.status
#cat ${OUT_PATH}/q.status
HAS_NONE_ZERO=`grep -v " 0" q.status`

if [ ! -z "${HAS_NONE_ZERO}" ]
#if [ -z "${HAS_NONE_ZERO}" ]
then
    tableit ${OUT_PATH}/q.status |/sherif/mail_alert2.sh "RabbitMQ Alert"
fi
The above will provide output only if  any of the queues has messages in them.
More sophisticated NodeJS or shell scripts can be built around these.
The above tabelit function is added for more cool email look :)
using poweful formatting of C like printf shell function.
Also the mail_alert2.sh uses shell piping as input, details on this found on another post.