Apache 2.2.x mod proxy balancer issue with hung http back ends

This came to me a a surprise!!!

It seems when a tomcat back end hangs, apache mod_proxy_balancer is unable to detect it is not working, while if the node it shutdown, it is able to see it is not there and thus stops sending requests to it.

Hung backend however keep getting requests and it consumes all the apache resources and slows down the given site to a huge extent !

I have managed to regenerate the issue locally on my Vbox.

Just run any balanced 2 or more tomcats behind apache and send a kill –STOP to one of them.

This will keep tomcat running (its http port accepting connections) but seems to apache as if it is hung since it is stopped and doing nothing.

Apache proxy balancer didn’t have a clue about this; and starts to get very slow responses and proxy errors.

To resume tomcat, send kill -CONT to it.

Shutting down tomcat, allows proxy balance to see that there is no connections accepted and it stops sending requests to this shutdown nodes successfully.

So we need to check if we can tweak the config of proxy balance to detect hung backends and stop ending requests to them . .

I hope we could do that using proxy balancer !!

I think mod_jk might not have this issue . . but that would be a bit of a change since in all our application we use proxy balancer . .

http://grokbase.com/t/tomcat/dev/039pt6n1zz/mod-jk-does-not-detect-a-hung-tomcat

Still, mod_jk would be a better option since uses ajp; a binary protocol; offers more security and better performance than plan http based proxy balancer.

  . . . . . . . . . . .. . . . . . .

Further reading into this, the issue is fixed in apache 2.4.x.

Using the ping parameter in 2.4.x is supported for both AJP and HTTP . .

We can’t use apache 2.4.x since it is not supported by RSA access manager and RSA agent as of now, so i need to do something else !

I tested AJP proxy balancing and this issue is not there when using ping option.

We just need to configure the AJP connector on TC have connection timeout set to some reasonable value and it is working fine.

Config looks like this:

apache Proxy balancer:

<IfModule mod_proxy.c>
    ProxyRequests Off
    <Proxy *>
        AddDefaultCharset off
        Order deny,allow
    </Proxy>
    ProxyVia On
    ProxyPreserveHost On
    <Proxy balancer://mycluster>
        #BalancerMember http://localhost:8080 min=300 smax=1000 route=n8080
        #BalancerMember http://localhost:8081 min=300 smax=1000 route=n8081
        BalancerMember ajp://localhost:8009 min=300 smax=1000 route=n8080 ping=100ms
        BalancerMember ajp://localhost:9009 min=300 smax=1000 route=n8080 ping=100ms
    </Proxy>
    ##### APPS PROXY #####
     ProxyPass /iamup.html !
    ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=on
    ProxyPassReverse / balancer://mycluster/
</IfModule>

Tomcat server.xml:

<!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443"
        connectionTimeout="20000"/>
 

<!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
    -->
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="n8081">
 

 

 Cool stuff, Apache never stop amazing me :)