Stopping Apache 2.4 SSL proxy from chekcing backend Certificate

This post is intended for testing purposes only.
On production, apache should be allowed to check the cert as it is supposed to do by default.
This is one of the good hardening features that apache 2.4.x has over apache 2.2.x.

To stop all SSL validation on doing a proxy to an HTTPS back end; we need to use the following directives:

SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off

as of apache 2.4.7 the directive SSLProxyCheckPeerName overrides the PeerCN one and is also needed to be set to off.

Please check the below Stackoverflow link.
http://stackoverflow.com/questions/19294816/is-it-possible-to-ignore-an-apache-proxyd-certificate

Rundeck example job definition

The below is a rundeck example job definition that uses options inside a script.

<joblist>
  <job>
    <id>3f6b1581-9b1d-4477-9980-627a95c0d676</id>
    <loglevel>INFO</loglevel>
    <sequence keepgoing='false' strategy='node-first'>
      <command>
        <scriptargs>${option.args_1}</scriptargs>
        <script><![CDATA[#Inline.sh
#!/bin/bash
echo "got this from the Job: @option.args_1@"
cat /dev/null >/tmp/${HOSTNAME}_rd_tst
echo "#inserting value from the RD job" >>/tmp/${HOSTNAME}_rd_tst
if [ "x${1}" = "x" ]
then
    echo "testid=100" >>/tmp/${HOSTNAME}_rd_tst
else
    echo "testid=${1}" >>/tmp/${HOSTNAME}_rd_tst
fi
echo "executed on ${HOSTNAME} at `date`"
echo "done"]]></script>
      </command>
    </sequence>
    <description>This job will create a file named ${HOSTNAME}_rd_tst on a remote node on temp folder and put in a key and default value of testid=100.</description>
    <name>Simple_file_key_value</name>
    <context>
      <project>Test_remote</project>
      <options>
        <option name='args_1' value='100' required='true' />
      </options>
    </context>
    <dispatch>
      <threadcount>1</threadcount>
      <keepgoing>false</keepgoing>
      <excludePrecedence>false</excludePrecedence>
      <rankOrder>ascending</rankOrder>
    </dispatch>
    <uuid>3f6b1581-9b1d-4477-9980-627a95c0d676</uuid>
    <nodefilters>
      <filter>beren.*</filter>
    </nodefilters>
  </job>
</joblist>

Rundeck API small script

This is a demo script for using Rundeck and its API.

Rundeck API is a cumulative work that has api numbers from 1 to 12 up to now.
It will allow you to list and execute rundeck jobs remotely and also to monitor them remotely.
This will be very useful also for rundeck scripts that would call another remote rundeck jobs, this can be a good work around for remote execution when SSH is not allowed for security reasons.
Also would server to put your entire Ops jobs and infra under control using a simple web application.

The applications for this is limitless.
link this with my earlier post for nodeJS shell execution and u will have a lot of options to automation.

Below is a small example:

PROJECTS=`curl -H "Accept: application/json" http://beren:4440/api/1/projects?authtoken=It4kb1tIgw8gBpKI3rQhscKTp0OVYSNo |tr "}" "\n"|tr "," "\n"|grep name|cut -d":" -f2 |tr -d "\""`

#echo $PROJECTS
for proj in $PROJECTS
do
        #get all Jobs in all projects:
        echo "Project: $proj"
        curl "http://beren:4440/api/1/jobs?authtoken=It4kb1tIgw8gBpKI3rQhscKTp0OVYSNo&project=${proj}" #egrep "\<name>|\<job id="
done

To Run your job directly use a URL similar to below:

curl "http://beren:4440/api/1/job/61894b63-d687-4a39-a583-e0eba070339f/run?authtoken=It4kb1tIgw8gBpKI3rQhscKTp0OVYSNo"

Systemd vs SysV Linux services

Check the below commands for systemd new config:

[root@localhost systemd]# systemctl disable firewalld.service
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
[root@localhost systemd]#

[root@localhost systemd]# systemctl enable firewalld.service
ln -s '/usr/lib/systemd/system/firewalld.service' '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
ln -s '/usr/lib/systemd/system/firewalld.service' '/etc/systemd/system/basic.target.wants/firewalld.service'
[root@localhost systemd]#


[root@localhost systemd]# systemctl stop firewalld.service
[root@localhost systemd]#

[root@localhost systemd]# systemctl disable firewalld.service
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'
[root@localhost systemd]#

[root@localhost systemd]#  systemctl list-unit-files -t service ^C

of F5 BigIP and Session stickiness

F5 BigIP is well know for its robustness and superior performance in the the Loadbalancer business.

I was exposed to using it in a architecture similar to the one described below:
http://www.f5.com/pdf/deployment-guides/tomcat-dg.pdf

Actually, F5 would better serve as SSL offloading step and will mask the underlying Apache and Tomcat instances.

Two things needs to be considers:
1- Since we have 2 LB VIPs, each of them should maintain a separated cookie for session stickiness. this is to avoid one overwriting another.
2- If your app contains a lot of Ajax and asyncronous calls, OneConnect needs to be enabled on the Tomcat LB, so that the session stickiness decision is processed correctly for each request. this is becasue BigIP does the load balancing on the TCP connection rather than request basis.

Apache 2.2.x mod proxy balancer issue with hung http back ends

This came to me a a surprise!!!

It seems when a tomcat back end hangs, apache mod_proxy_balancer is unable to detect it is not working, while if the node it shutdown, it is able to see it is not there and thus stops sending requests to it.

Hung backend however keep getting requests and it consumes all the apache resources and slows down the given site to a huge extent !

I have managed to regenerate the issue locally on my Vbox.

Just run any balanced 2 or more tomcats behind apache and send a kill –STOP to one of them.

This will keep tomcat running (its http port accepting connections) but seems to apache as if it is hung since it is stopped and doing nothing.

Apache proxy balancer didn’t have a clue about this; and starts to get very slow responses and proxy errors.

To resume tomcat, send kill -CONT to it.

Shutting down tomcat, allows proxy balance to see that there is no connections accepted and it stops sending requests to this shutdown nodes successfully.

So we need to check if we can tweak the config of proxy balance to detect hung backends and stop ending requests to them . .

I hope we could do that using proxy balancer !!

I think mod_jk might not have this issue . . but that would be a bit of a change since in all our application we use proxy balancer . .

http://grokbase.com/t/tomcat/dev/039pt6n1zz/mod-jk-does-not-detect-a-hung-tomcat

Still, mod_jk would be a better option since uses ajp; a binary protocol; offers more security and better performance than plan http based proxy balancer.

  . . . . . . . . . . .. . . . . . .

Further reading into this, the issue is fixed in apache 2.4.x.

Using the ping parameter in 2.4.x is supported for both AJP and HTTP . .

We can’t use apache 2.4.x since it is not supported by RSA access manager and RSA agent as of now, so i need to do something else !

I tested AJP proxy balancing and this issue is not there when using ping option.

We just need to configure the AJP connector on TC have connection timeout set to some reasonable value and it is working fine.

Config looks like this:

apache Proxy balancer:

<IfModule mod_proxy.c>
    ProxyRequests Off
    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
    </Proxy>
    ProxyVia On
    ProxyPreserveHost On
    <Proxy balancer://mycluster>
        #BalancerMember http://localhost:8080 min=300 smax=1000 route=n8080
        #BalancerMember http://localhost:8081 min=300 smax=1000 route=n8081
        BalancerMember ajp://localhost:8009 min=300 smax=1000 route=n8080 ping=100ms
        BalancerMember ajp://localhost:9009 min=300 smax=1000 route=n8080 ping=100ms
    </Proxy>
    ##### APPS PROXY #####
     ProxyPass /iamup.html !
    ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=on
    ProxyPassReverse / balancer://mycluster/
</IfModule>

Tomcat server.xml:

<!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
        connectionTimeout="20000"/>
 

<!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="n8081">
 

 

 Cool stuff, Apache never stop amazing me :)

 

 

 

Java classes in heap histo

below is a small table to decode class names like [C, [B and [Lclassnames:

Element Type        Encoding
boolean             Z
byte                B
char                C
class or interface  Lclassname;
double              D
float               F
int                 I
long                J
short               S 

This came in handy from stack overflow :)