JMX on TC Server (Tomcat)

Currently working on a PoC to enable remote JMX on TC server.

The config to do this goes into 3 places:
1- Setenv.sh:
needs to put in JVM parameters to allow JMX remotely using authentication

JMX_OPTS="-Dcom.sun.management.jmxremote
  -Dcom.sun.management.jmxremote.port=16001
  -Dcom.sun.management.jmxremote.ssl=false
  -Dcom.sun.management.jmxremote.authenticate=true
  -Dcom.sun.management.jmxremote.password.file=${CATALINA_BASE}/conf/jmxremote.password
  -Dcom.sun.management.jmxremote.access.file=${CATALINA_BASE}/conf/jmxremote.access"

2- jmxremote.access:
[root@khofo05 conf]# cat jmxremote.access
#admin readonly
admin readwrite
[root@khofo05 conf]#

3- jmxremote.password
[root@khofo05 conf]# cat jmxremote.password
# The "admin" role has password "springsource".
admin springsource
[root@khofo05 conf]#

The above is sufficient to have an authenticated remote JMX up and running on any tomcat.

I wanted to explore using SSL, for more protection since JMX would allow altering server parameters if readwrite rule is required, but since I need JMX  for monitoring purpose is would just need to have read only rules.

After some reading and consulting my colleges i decided to abandon JMX and go with Jstatd instead.

AWK string manipulations

The below script is used to scan a full text file line by line for a certain pattern.
Once this pattern is found,
The script does another pattern matching to do another change in that same line.

[root@khofo05 sherif]$  awk -v FILENAME=./server.xml_20131127_1 -f ./change_ds.awk >server.xml
[root@khofo05 sherif]$ cat change_ds.awk
BEGIN{
        i=0;
        line="";
        while (getline line < FILENAME)
        {
                gotit=match(line,"DB1");
                if (gotit == 0)
                {
                        print(line);
                }
                else if (gotit > 0 )
                {
                        #print ("match------",line);
                        newsvr=gsub("db111oracle.example.com","db123oracle.example.com",line);
                        newsid=gsub("DB1","DB3",line);
                        #print (newsvr,newsid,line);
                        print (line);
                        #break;
                }
        }
}
[root@khofo05 sherif]$


The above is an example for a mass change in a server.xml file for changing a Datasource definition from a SID/host to another SID/host.
Would be useful if multiple datasources are involved.

Extracting a part of the JVM startup command

This  One is used to extract the Catalina base directory from a running TC server process.
This is useful for monitoring what applications are being deployed on those TC instances.

 
[root@khofo05 ~]# ps -ef |grep java|grep 18001
root     20342     1  0 Jan20 ?        00:01:05 /apps/admngop1/springsource/jdk1.6/bin/java -Djava.util.logging.config.file=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/khofo05_18001/conf/logging.properties -Xmx512M -Xss192K -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=16001 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.password.file=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/khofo05_18001/conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/khofo05_18001/conf/jmxremote.access -Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager -Djava.endorsed.dirs=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/tomcat-6.0.35.A.RELEASE/endorsed -classpath /apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/tomcat-6.0.35.A.RELEASE/bin/bootstrap.jar -Dcatalina.base=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/khofo05_18001 -Dcatalina.home=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/tomcat-6.0.35.A.RELEASE -Djava.io.tmpdir=/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/khofo05_18001/temp org.apache.catalina.startup.Bootstrap start


[root@khofo05 ~]# JAVA_CMD=`ps -ef |grep java|grep 18001`
[root@khofo05 ~]# CATALINA_BASE=`echo ${JAVA_CMD##*Dcatalina.base}|cut -d "=" -f2|cut -d" " -f1`

[root@khofo05 ~]# echo $CATALINA_BASE
/apps/admngop1/springsource/vfabric-tc-server-standard-2.6.3.RELEASE/khofo05_18001
[root@khofo05 ~]#

This would allow to check the webapps folder, the conf and others.
Thought to document the bash string concat constructs, never used it before :)

Running a Shell script from an HTML page using NodeJs as backend

The below code is a modified version from the one used for PHP on this link: http://www.scriptol.com/javascript/nodejs-php.php

This functionality could be useful in doing many automation tasks for Ops, though could be done by other tools like Puppet or even Jenkins but NodeJs is very light weight and quite fast.
Would plan to integrate this in my HTML status pages in the future.

The code:

var sys = require("sys"), 
http = require("http"),   
path = require("path"),
url = require("url"),
filesys = require("fs"),
runner = require("child_process");

function sendError(errCode, errString, response)
{
  response.writeHead(errCode, {"Content-Type": "text/plain;charset=utf-8"});
  response.write(errString + "\n");
  response.end();
  return false;
}

function sendData(err, stdout, stderr, response)
{
  if (err) return sendError(500, stderr, response);
  response.writeHead(200,{"Content-Type": "text/plain;charset=utf-8"});
  response.write(stdout);
  response.end();
}

function runScript(exists, file, response)
{
  if(!exists) return sendError(404, 'File not found', response);
  runner.exec(file ,
   function(err, stdout, stderr) { sendData(err, stdout, stderr, response); });
}

function myshell(request, response)
{
  var urlpath = url.parse(request.url).pathname;
  //var param = url.parse(request.url).query;
  var localpath = path.join("/", urlpath);
  console.log(localpath);
  filesys.exists(localpath, function(result) { runScript(result, localpath, response)});
}

var server = http.createServer(myshell);
server.listen(7071);
console.log("Shell ready to run script given on port 7071.");


HTML:

<html>

<head>
<meta http-equiv="refresh" content="600" />
</head>
               <body>
                <br>
                <a href=http://localhost:7071/bin/ls>Run ls Locally</a>
                <br>
                <br>
        </body>
</html>

Jstatd configuration for JVM monitoring

Currently I am busy setting up configuration for running Jstatd on a multiple set of VMs runing Spring source TC servers.
The setup is simple, use a script to start the jstatd locally and invoke the script from a master VM to start on all nodes.
Jstatd is useful for memory monitoring for performance testing of Java applications and can provide realtime info about the JVM execution.

In our case Jstatd is much better than using traditional JMX parameters on JVM, JMX would need to be secured and have a strong authentication and authorization in place to secure the application.
This can be done, but the amount of config changes wouldn't be easily done unless we have a Dev-Ops tool like puppet to handle.

Jstatd is zero config from application point of view, just put the policy file and start the deamon and ur done !!

To start Jstatd on a server you need a policy file: jstatd.all.policy
 That contains:
grant codebase "file:${java.home}/../lib/tools.jar" { permission java.security.AllPermission; };

and just run as:

JAVA_HOME/bin/jstatd -J-Djava.security.policy=jstatd.all.policy

Jstatd uses port 1099 (RMI) to expose JVM info like memory and thread usage.
Jstatd also exposes all JVMs running on the same server using one jstatd process.

We can use jvisualvm tool provided by Java SDK to connect to remote Jstatd and browse the needed info.

Web based status Page

Just finished creating a set of shell scripts to create dynamic web based status pages for Support Zone application.
The requirement is to have a full map of what web applications are deployed and which services are being active.
The page updates every 10 mins by a shell script that runs on crontab; the script regenerates the html pages dynamically by checking the services based on a set of config files that holds what needs to be checked.

below how it looks like:

The tables above are iframes that show the html generated by the shell scripts.
if the port number is shown in red, it means the service is not running on this port, could be that a service is down or the config needs to be updated.
Also there is email alerting functionality available in case something seems to be wrong.

The checks are being done by curl pinging web interfaces of those services, if the response didn't come in a defined amount of time (20 sec) the port will show as red.

Those scripts can be used dynamically for checking any web / internet  based service that listens on a socket  . . .
Currently vfabric Hyperic monitoring tool doesn't offer a similar summary page, so those custom scripts were created to fill in this gap.
Other tools can offer similar or superior interfaces like HP BAC /Sitescope or CA Intrascope.
Those tools offer performance information and deeper insite on business transactions and internal JVM operation.



Automation is good  :)

SQLFire CLI

We have been doing manual SQLFire deployment ever since it was enrolled as a caching solution in the Service Support Zone project.

Part of the deployment is to do standard war file deployment on TC server, which is easy to automate.
The issue was with running SQL queries that needed to be run at various stages before and after the deployment.
SQLFire doesn't offer a standalone client like Oracle, we need to write SQL scripts and pass it to the comand line interface script: sqlf.
The command runs as follows:

${SQLFIRE_HOME}/bin/sqlf run -client-bind-address=${NODE} -client-port=${PORT} -user=${USERNAME} -password=${PASSWORD} -file=${SQL_SCRIPT}

I am exploring how this simple command can be used to run remotely from a Jenkins job to support fully automated SQLFire deployment.

Managed to use the command for doing table backups :)
But faced an issue with running this from crontab, needs to have all the environment variables defined before it runs so i had to issue the command using bash -l from within cron.

Collecting System information from 230+ VMs

Today I was working on collecting system info from 230+ VM composing the entire environment of the EMC Service Support Zone project.
Those proved to be a bit of challenge since many of them didn't have SSZ keys installed on the control server and I had to do manually :)
Though, would be a lot of help if this can be accomplished as it would server all future automation .
Below is a link on how to setup SSH keys:
https://www.digitalocean.com/community/articles/how-to-set-up-ssh-keys--2


Though Puppet agent would have abstracted all this if it is installed on the VMs.
Will need to see how this can be accomplished.

This task would also help automating vfabric SQLFire  deployment.
Will talk about this later.