Showing posts with label cookie. Show all posts
Showing posts with label cookie. Show all posts

Thursday, 15 September 2016

Apache / haproxy large Cookie size issues

Although HTTP protocol doesn't have a limit on the cookie size, which is in turn a part of the request/response headers, Apache does impose a limit to protect the webserver from denial of service attaches.

Apache controls this using the "LimitRequestFieldSize" directive which defaults to 8192 byte as a max for a header field.

If larger cookies are needed, we need to bump up this value to a bigger number.
eg:

LimitRequestFieldSize 18192

in order to test this, i set up a simple webserver config with the below config:

RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match

LimitRequestFieldSize 18192

Header set Set-Cookie "apachenode=node1; path=/;"
Header add Set-Cookie "env=test; path=/;"
Header add Set-Cookie "TimeS=NONE; path=/;"


The LimitRequestFieldSize only works on Requests not on the response, thus apache can set-Cookies with large values with no problems, its the job of the browser to validate this.

To be able to test this from Firefox, I used the Cookie Manager plugin https://addons.mozilla.org/en-US/firefox/addon/cookies-manager-plus/ to set a large cookie that is 24k in size with a random string generated from the site: http://textmechanic.com/text-tools/randomization-tools/random-string-generator/

I managed to prove the compiled limit in Centos 7 is indeed 8192 and expanded that to ~18k as seen above and it worked.

When apache fails to accept the request it responses with an http 400 error as below:


I tried to do the same test with commandline curl but seems curl truncates the large cookie when using the -b option and passing a file as below:

[root@feanor conf]# curl -b ./cookies.txt  http://localhost/new.html
this is a new page :)
[root@feanor conf]# ls -ltrh ./cookies.txt
-rw-r--r-- 1 root root 25K Sep 15 12:28 ./cookies.txt
[root@feanor conf]#


Although the cookie file contains a single cookie that is 24k in size, curl seems to have truncated it and the request went in. 
lowering the LimitRequestFieldSize to something like 4600 managed to have curl reproduce the same behaviors as the browsers:

[root@feanor conf]# curl -b ./cookies.txt  http://localhost/new.html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Size of a request header field exceeds server limit.<br />
<pre>

</pre>
</p>
</body></html>
[root@feanor conf]#

Thus, curl could be tricking you if you are debugging an issue like this from the command line.

Also one more note about large Cookie sizes, if haproxy is used in the setup, eg: as a balancer in front of Apache or for SSL offloading, it could be needed to increase the tune.bufsize so that it can accept larger requests and larger Cookie sizes.

Haproxy checks the size as (tune.bufsize - tune.maxrewrite) where Maxrewrite is adjusted to half buffersize if that is larger than 1024.
Given the above info, tune.bufsize should be set to be double the Apache LimitRequestFieldSize.
Watch out for memory and ensure haproxy has enough resources to work with no issues.



Monday, 11 July 2016

HAproxy configuration as an HTTP router / balancer

HAproxy is a very flexible reliable TCP proxy and balancer middleware.
It can be used as a generic TCP proxy / port mapper or as a TCP load balancer.
It also supports using HTTP protocol mode where it is able to work as an http proxy server and loadbalancer.

In this post i will focus on the http mode, it is used more for implementing web proxies providing high availability for web applications. below is a sample config for HAproxy as an http proxy and router:


#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#   http://haproxy.1wt.eu/download/1.3/doc/configuration.txt
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 
    maxconn     400000
    user        sherif
    group       sherif
    daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode        http
    log         global
    option      dontlognull
    option      httpclose
    option      httplog
    option      forwardfor
    option      redispatch
    timeout connect 100000 # default 10 second time out if a backend is not found
    timeout client 600000
    timeout server 600000
    maxconn     600000
    retries     3
    errorfile 503 /etc/haproxy/errors/503.http
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main vardamire:8080
acl app1_url                     path_beg     /app1
acl test_url                    path_beg    /tst
acl app2_url                    path_dir    app2
###################################################################
use_backend app1_8080                        if   app1_url
use_backend app2_8081                        if   test_url app2_url   
use_backend default_services
###################################################################
backend app1_8080
    balance     roundrobin
    option httpchk GET /app1/alive
    cookie JSESSIONID prefix nocache
    server      app1    server1:8080 check cookie a1
    server      app2    server2:8080 check cookie a2
    server      app3    server3:8080 check cookie a3 


backend app2_8081
    balance     roundrobin
    option httpchk GET /tst/app2/alive
    server      app1     server2:8081 check
    server      app2     server3:8081 check
 

backend default_services
   server app1 web.toplevelbalancer:8080 check

listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth admin:admin


The above config is an example for doing http routing and balancing based on URL patterens.
Also it shows how HAproxy can handle session stickiness.

The keyword acl defines a url pattern using either pathbeg (path begins) or path_dir (path directory portion).
Then the backend keyword is used to define a couple of application backends to HAproxy, those will do the actual serving of the content.
the balance keyword is used to tell HAproxy to do a round robin load balancing between the defined servers, also it adds the option httpchk which will do an http check on the given URI for each defined server to determine if it is up or not.
Also cookie keyward is used to append a part to the JSESSIONID cookie and have it checked by HAproxy to be able to maintain session stickiness. HAproxy will prefix JSESSIONID with the cookie defined in each server and thus will be able to keep track which session goes to which server.

Lastly we are enabling HAproxy statistics so that we can monitor the status of our backends and also the stats of the requests coming to them.

This config was used successfully to route and balance 40+ services for a big project and it working fairly smooth even under load testing.

HAproxy is very light weight and can handle 10s of thousands of connections without issues.