Java classes in heap histo

below is a small table to decode class names like [C, [B and [Lclassnames:

Element Type        Encoding
boolean             Z
byte                B
char                C
class or interface  Lclassname;
double              D
float               F
int                 I
long                J
short               S 

This came in handy from stack overflow :)

Some useful scriptets

This one is a one liner to collect memory info from a set of servers.
Uses the command seq -w which pads the numbers with leading 0 to follow naming convention.
is simpler than the form for ((i=1;i<16;i++)) which will not pad the numbers.

for i in `seq -w 01 16`;do echo "sheif_prd${i} Mem=  `ssh -q sherif@sheif_prd$i free|grep Mem|tr -s " " |cut -d " " -f2`" ; done

This uses to be useful to collect simple info like memory or CPU or such things.


Another one is for fixing SQLFire backups taken by sqlf -file option.
the backup is done using this script: 

RunSQLFire_sqlscript.sh:

source ${HOME}/.bash_profile

if [ "x${1}" = "x" ]
then
    echo "missing SQL script"
    exit 1
fi

${HOME}/sqlfire/current/bin/sqlf run -client-bind-address=${HOSTNAME} -client-port=1527 -user=prd -password=prd -file=${1}

Then backup is taken in the following format:

PRMPT-$ head table_data
sqlf> SELECT * FROM "PRD"."TABLE";
Part_number                   |Serial                        |Type                      |ALLOWED            
----------------------------------------------------------------------------------------------------
4028227                       |6542883                       |E                             |1           
11256276                      |14533312                      |P                             |0     
3798130                       |18648804                      |C                             |0          
14000630                      |574577                        |E                             |1         
2059762                       |18986693                      |E                             |1         


The below AWK one liners converts the above into SQL inserts:

cat table_data |tr -s " " |awk -F"|" '{print("\x27"  $1 "\x27" "," "\x27" $2 "\x27" "," "\x27" $3 "\x27" ","  $4  "," "{ts  \x27" $5 "\x27 }");}' |sed -e "s/ '/'/g" >table_data_fixed

Note the use of ASCII for " ' "  as it was difficult to print it from within AWK.

cat  table_data_fixed |sed -e's/^/INSERT INTO TABLE (Part_number,Serial,Type,ALLOWED) VALUES (/g' |sed -e 's/$/);/g' >table_data_inserts.sql


 

Lynx usage in shell scripts

Lynx proved to be a very useful tool for avoiding complex shell scripts around html based reports and logs.
check the below command:

/apps/index/DRE_analysis_scripts/lynx -cfg=/apps/index/DRE_analysis_scripts/lynx.cfg -dump "http://dreprd2:9000/?action=getrequestlog&openlinks=on&refresh=0&tail=1000" >${OUTDIR}/DRE20_${TSTAMP}.txt 

The above shell snippet was useful for analyzing autonomy idol DRE server web based search transaction logs.
Those logs are not dumped to file with same format and i liked the web based format more than the standard content component logging.

Lynx helped me use this elegant log format from command line and apply standard shell filters like grep, cut etc. to it.

Lynx proved to be very handy too :)

Converting variable base DS info into JNDI

This was quite an issue for me since a while.
There are 3 ways to put in the Data source info into a tomcat.
1- put it in context.xml directly which will save time and effort.
2- put into server.xml as global naming resources and then add an entry into context.xml for global resource link
3- put as variables into catalina.properties and then link them in server.xml and context.xml which is the longest way !

below is a script that changes the variable notation to the server.xml notation.
Also will print out the resource link for context.xml:

below is how the ds info got from catalina.properties look like:

[root@Beren DS]# head ds_info
aaa.conf.datasource1.name=configDataSource
aaa.conf.datasource1.url=jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS = (PROTOCOL = TCP)(HOST = orcl_host)(PORT = 1527))(CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = orcl)))
aaa.conf.datasource1.username=sherif
aaa.conf.datasource1.password=mydspassword

aaa.conf.datasource2.name=tstDataSource
aaa.conf.datasource2.url=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=orcl_host)(PORT=1700))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=test)))
aaa.conf.datasource2.username=test
aaa.conf.datasource2.password=test
[root@Beren DS]#

Below is the script:

[root@Beren DS]# cat fix.sh                             
#set -x                                                 
OLD_IFS=${IFS}                                          
IFS='                                                   
'                                                       
for line in `cat ds_info`                               
do                                                      
        VAR=`echo ${line}|cut -d"=" -f1|cut -d"." -f4`  
        VAL="`echo ${line}|cut -d"=" -f2-`"             

        if [ "${VAR}" = "name" ]
        then                   
                echo '<Resource name="jdbc/'"${VAL}"'"'
                echo 'auth="Container" type="javax.sql.DataSource"'
        fi                                                        
        if [ "${VAR}" = "url" ]                                   
        then
                echo 'url="'"${VAL}"'"'
                echo 'driverClassName="oracle.jdbc.driver.OracleDriver"'
                echo 'maxActive="80"'
                echo 'maxIdle="20"'
                echo 'maxWait="10000"'
        fi
        if [ "${VAR}" = "username" ]
        then
                echo 'username="'"${VAL}"'"'
        fi
        if [ "${VAR}" = "password" ]
        then
                echo 'password="'""${VAL}'"'
                echo 'removeAbandoned="true"'
                echo 'removeAbandonedTimeout="60"'
                echo 'logAbandoned="true"'
                echo 'validationQuery="SELECT SYSDATE FROM DUAL"/>'
                echo " "

        fi
        #read
done

echo ""
echo "Context.xml"
echo ""

for line in `cat ds_info|grep "emc.conf.datasource.*\.name"`
do
        VAR=`echo ${line}|cut -d"=" -f1|cut -d"." -f4`
        VAL="`echo ${line}|cut -d"=" -f2-`"

        if [ "${VAR}" = "name" ]
        then
                echo ' <ResourceLink name="jdbc/'${VAL}'" global="jdbc/'${VAL}'" type="javax.sql.DataSource"/>'
        fi
        #read
done
[root@Beren DS]#

Changing JVM options using a shell script

The below script is used to customize a TC sever setenv.sh for multiple TC instances.
The script reads a config file and applys this to the setenv and pushes it to the instace loaction.


[sherif@khofo05 ~ ]$ cat fixSetenv.sh
OLD_IFS=${IFS}
IFS='
'
for INST in `cat tc_nodes |grep -v "^#"`
do 
    USER=`echo ${INST} |cut -d":" -f1`
    TC_INST=`echo ${INST} |cut -d":" -f2|cut -d"@" -f1`
    HOST=`echo ${INST} |cut -d":" -f2| cut -d "_" -f1`
    HTTP_PORT=`echo ${INST} |cut -d "_" -f2|cut -d"@" -f1`
    HOME_DIR=`echo ${INST} |cut -d"@" -f2`
    JVM_PAR=`echo ${INST} |cut -d"@" -f3`
    NEW_JAVA_HOME="JAVA_HOME=${HOME_DIR}/springsource/jdk1.7.0_21"

     OLD_JVM_PAR=`grep "JVM_OPTS=" setenv.sh|tr -d '\n'`
    OLD_JAVA_HOME=`grep "JAVA_HOME=" setenv.sh|tr -d '\n'`
IFS=${OLD_IFS}
#echo "---------------"
echo "working on ${HOST}"
echo "sed -- 's%${OLD_JVM_PAR}%${JVM_PAR}%g' setenv.sh |" >sed_cmd
echo "sed -- 's%${OLD_JAVA_HOME}%${NEW_JAVA_HOME}%g'" >>sed_cmd
echo "creating custome setenv "
bash ./sed_cmd >setenv.sh_${TC_INST}
scp setenv.sh_${TC_INST}  ${USER}@${HOST}:${HOME_DIR}/springsource/vfabric-tc-server-standard-2.9.2.RELEASE/${TC_INST}/bin/setenv.sh
echo "sent to ${TC_INST}
Done"
done
IFS=${OLD_IFS}
[sherif@khofo05 ~ ]$

The instances config file is named tc_nodes.
This file looks like this:

username1:hostname01_8080@/home/dir/of/username1@JVM_OPTS="-server -Xms4096m -Xmx4096m -Xss192k -XX:MaxPermSize=512m -XX:PermSize=256m -XX:ParallelGCThreads=2 -XX:NewSize=256m -XX:MaxNewSize=256m -Xnoclassgc -XX:SurvivorRatio=14 -Xloggc:${HOME}/logs/gc.log"

This is parsed for the parameters used by the script.
I found it more difficult to run the sed commads from within the same script. so to solve this the fixsetenv.sh creates another script and runs it.
this is the sed_cmd script.
this one looks like this:

[sherif@khofo05 ~]$ cat sed_cmd
sed -- 's%JVM_OPTS="-Xmx512M -Xss256K"%JVM_OPTS="-server -Xms4096m -Xmx4096m -Xss192k -XX:MaxPermSize=512m -XX:PermSize=256m -XX:ParallelGCThreads=2 -XX:NewSize=256m -XX:MaxNewSize=256m -Xnoclassgc -XX:SurvivorRatio=14 -Xloggc:${HOME}/logs/gc.log"%g' setenv.sh |
sed -- 's%JAVA_HOME="/homedire1/springsource/jdk1.7.0_21"%JAVA_HOME=/homedire1/springsource/jdk1.7.0_21%g'
[sherif@khofo05 ~]$

The above made it possible to do custom changes based on the config files.
In order to come around parsing the whole JVM_OPTs with the spaces and things like that I had to change the Internal field separator.
This made it possible to accommodate the spaces.


Automation is fun :)

SSL Server cert import for Java apps

The below is to document the procedures to import Server certs into the JVM.
This is to avoid exceptions like the below one:


Exception Message: sun.security.validator.ValidatorException: PKIX path building failed

The above is specifically showing an issue with the certification path that should be included in the JVM cacerts.
For info about the JVM certificate files please check this URL :
http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManager

Steps :

1- Acquire the needed server certificate from the server URL if possible by exporting it from Firefox.
Please make sure you are getting all the certificate path exported.
This would mean that you may need to export root cert, intermediate cert and the leaf server cert.
Would be better to obtain the cert from the server support team, would be much better if possible.

2-Import Certs:
 /opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -import -trustcacerts -v -alias RSAV2 -file /tmp/RSAV2.cer

The above inserts an intermediate cert signed by RSA cert. authority and importing it with the user defined alias  RSAV2.

You might need to import all the certs defined in the server cert path.

3- List Certs:
/opt/jdk1.6.0_26/bin/keytool -keystore cacerts -storepass changeit -list
You should be able to see the certs that you have imported in the list along with the date the cert was imported.

 Also below is a very useful link for doing SSL cert and key debuging in case you are setting up the server and creating a certificate.

http://www.sslshopper.com/article-most-common-openssl-commands.html

Clustering issues on SQLFire and RabbitMQ

I have been seeing much clustering issues in the last months on both RabbitMQ and SQLFire and both are Pivotal products which are opensource.
Seems like both products have issues with network latency that would cause Split brain issues in the cluster and could lead to potential data loss.

In order to be able to tell when such issues happen i have used the following approches to be able to tell if we have a cluster issue:

1- Integrate Hypric monitoring with SQLFire & RabbitMQ components.
2- For SQLFire, we can make use of the following system query:

 cat get_members.sql
select ID,KIND from sys.members order by KIND;

Running this query with commandline :
{HOME}/sf/sqlf run -client-bind-address=${HOSTNAME} -client-port=1527 -user=myapp -password=myapp -file=get_members.sql

Parsing this output would allow knowing the current number of cluster members.
if any split happens the output of this query will be differant.

3- For RabbitMQ, used a more radical way to do the monitoring.
RabbitMQ nodes will be always talking to each other, so the warning is based on the number of connections that each node has towords the sister node in the cluster:

    CON_COUNT=`ssh -q rmquser@rmqnode01 netstat -p 2>/dev/null|grep -i est |tr -s " "|cut -d" " -f5,7|grep rmqnode|cut -d"." -f1,4 --output-delimiter=" "|cut -d" " -f1,3 |sort |uniq -c|wc -l`

This will get the number of connections from rmqnode01 to all other cluster members.
The count should be number of clustermembers - 1

If the number is less, then we have a split brain issue.
Also  RabbitMQ management console tell you at once that there is an issue.

A future thing is to capture the warning from the RabbitMQ managment console directory.