Haproxy CSV stats

Haproxy exposes a very useful statistics page that can be accessed over a web browser or from the command line using a tool like curl.
Haproxy allows the stats to be exposed in a CSV format that is exceedingly useful if you are going to have a script around it.

To access the CSV stats from http interface use the below:

curl  -u statsuser:statspassword 'http://haproxyhost:8001/stats/?stats;csv;' 2>/dev/null |egrep -iv "up|^#|open" |cut -d"," -f1,2,18

Where port 8001 is the statistics port as defined in the haproxy config.
The config is simple, already shown in older posts, should be something like this:

listen stats *:8001
    stats enable
    stats uri /stats
    stats hide-version
    stats auth statsuser:statspassword


The above mini script will print out only the apps and backends that show as down.
working ones will not show up.
The CVS header is also stripped as the header always starts with a "#" this makes it possible to process only the data using a grep -v as above.

The CSV result contains a big amount of info that can be used for load management and automation.
The details of the haproxy CSV stats layout can be found at:
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#9  

Updating a field of a DB table using records from same table

The below query is to update a field that is set to null in a config table from records that contain values in the same table.

I came across this scenario while working on a puppet upgrade project where we are moving the file based Yaml configs to SQL DB.


below is the query:

update config_dev c

set groupname = l.groupname

from ( select distinct (groupname) , name from config_dev where groupname is not null ) l

where c.groupname is null

and c.name = l.name

The table contains some "groupname" fields that contains null, still those should contain same group values as others given that name field is the same.

the above query will do that for us.
Not that the above query will not run in MySQL !!

Thanks to my friend Mohamed Youssef who helped me with this query.

Setting up a new PostgreSQL on Redhat systems

Postgres setup turns out to be not as straight forward as I assumed.
When you setup the DB server using yum, the DB comes in a not initialized state, and thus it fails to start using standard service commands.

The below Fedora project wiki link entails a lot of details about this issue:
https://fedoraproject.org/wiki/PostgreSQL

An example on CentOS 7 looks like below.
Once you initialize the DB it starts up with no issues.

[root@vardamir ~]# systemctl start postgresql
Job for postgresql.service failed because the control process exited with error code. See "systemctl status postgresql.service" and "journalctl -xe" for details.
[root@vardamir ~]# journalctl -xe
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 3 has been created for the user root.
--
-- The leading process of the session is 11388.
Sep 19 11:54:29 vardamir systemd[1]: Starting Session 3 of user root.
-- Subject: Unit session-3.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-3.scope has begun starting up.
Sep 19 11:54:29 vardamir sshd[11388]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 19 11:54:30 vardamir dbus[725]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Sep 19 11:54:30 vardamir dbus-daemon[725]: dbus[725]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Sep 19 11:54:30 vardamir dbus[725]: [system] Successfully activated service 'org.freedesktop.problems'
Sep 19 11:54:30 vardamir dbus-daemon[725]: dbus[725]: [system] Successfully activated service 'org.freedesktop.problems'
Sep 19 11:54:52 vardamir polkitd[1394]: Registered Authentication Agent for unix-process:11426:31997 (system bus name :1.28 [/usr/bin/pkttyagent --notify-fd 5 --fallback],
Sep 19 11:54:52 vardamir systemd[1]: Starting PostgreSQL database server...
-- Subject: Unit postgresql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql.service has begun starting up.
Sep 19 11:54:52 vardamir postgresql-check-db-dir[11431]: "/var/lib/pgsql/data" is missing or empty.
Sep 19 11:54:52 vardamir postgresql-check-db-dir[11431]: Use "postgresql-setup initdb" to initialize the database cluster.
Sep 19 11:54:52 vardamir postgresql-check-db-dir[11431]: See /usr/share/doc/postgresql-9.2.15/README.rpm-dist for more information.
Sep 19 11:54:52 vardamir systemd[1]: postgresql.service: control process exited, code=exited status=1
Sep 19 11:54:52 vardamir systemd[1]: Failed to start PostgreSQL database server.
-- Subject: Unit postgresql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql.service has failed.
--
-- The result is failed.
Sep 19 11:54:52 vardamir systemd[1]: Unit postgresql.service entered failed state.
Sep 19 11:54:52 vardamir systemd[1]: postgresql.service failed.
Sep 19 11:54:52 vardamir polkitd[1394]: Unregistered Authentication Agent for unix-process:11426:31997 (system bus name :1.28, object path /org/freedesktop/PolicyKit1/Authe
[root@vardamir ~]# postgresql-setup initdb
Initializing database ... OK

[root@vardamir ~]# systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-09-19 11:58:57 EDT; 10s ago
  Process: 11542 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 11537 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 11546 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─11546 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─11547 postgres: logger process  
           ├─11549 postgres: checkpointer process  
           ├─11550 postgres: writer process  
           ├─11551 postgres: wal writer process  
           ├─11552 postgres: autovacuum launcher process  
           └─11553 postgres: stats collector process  

Sep 19 11:58:56 vardamir systemd[1]: Starting PostgreSQL database server...
Sep 19 11:58:57 vardamir systemd[1]: Started PostgreSQL database server.
[root@vardamir ~]#

To allow the postgresql server (postmaster) to listen on all interfaces, we need to pieces of config changes, 1 in postgresql.conf and other in pg_hba.conf as below.
in postgresql.conf we need to set :

listen_addresses = '*' 

so that the postmaster server listens on all interfaces.
second we need to add a host entry in pg_hba.conf so that clients are allowed to connect.
this config will depend on network config of the installed DB, an example is shown below:

host    all             all             10.0.0.0/8              md5

md5 is used to allow encrypted password authentication.
Don't forget to allow the Linux firewall to open port 5432 so that external users can access the newly created DB server.

Its always better to create another user than the default postgres user and use that for the DB work remotely.

Apache / haproxy large Cookie size issues

Although HTTP protocol doesn't have a limit on the cookie size, which is in turn a part of the request/response headers, Apache does impose a limit to protect the webserver from denial of service attaches.

Apache controls this using the "LimitRequestFieldSize" directive which defaults to 8192 byte as a max for a header field.

If larger cookies are needed, we need to bump up this value to a bigger number.
eg:

LimitRequestFieldSize 18192

in order to test this, i set up a simple webserver config with the below config:

RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match

LimitRequestFieldSize 18192

Header set Set-Cookie "apachenode=node1; path=/;"
Header add Set-Cookie "env=test; path=/;"
Header add Set-Cookie "TimeS=NONE; path=/;"

The LimitRequestFieldSize only works on Requests not on the response, thus apache can set-Cookies with large values with no problems, its the job of the browser to validate this.

To be able to test this from Firefox, I used the Cookie Manager plugin https://addons.mozilla.org/en-US/firefox/addon/cookies-manager-plus/ to set a large cookie that is 24k in size with a random string generated from the site: http://textmechanic.com/text-tools/randomization-tools/random-string-generator/

I managed to prove the compiled limit in Centos 7 is indeed 8192 and expanded that to ~18k as seen above and it worked.

When apache fails to accept the request it responses with an http 400 error as below:

 

I tried to do the same test with commandline curl but seems curl truncates the large cookie when using the -b option and passing a file as below:

[root@feanor conf]# curl -b ./cookies.txt  http://localhost/new.html
this is a new page :)
[root@feanor conf]# ls -ltrh ./cookies.txt
-rw-r--r-- 1 root root 25K Sep 15 12:28 ./cookies.txt
[root@feanor conf]#

Although the cookie file contains a single cookie that is 24k in size, curl seems to have truncated it and the request went in. 

lowering the LimitRequestFieldSize to something like 4600 managed to have curl reproduce the same behaviors as the browsers:

[root@feanor conf]# curl -b ./cookies.txt  http://localhost/new.html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Size of a request header field exceeds server limit.<br />
<pre>

</pre>
</p>
</body></html>
[root@feanor conf]#

Thus, curl could be tricking you if you are debugging an issue like this from the command line.

Also one more note about large Cookie sizes, if haproxy is used in the setup, eg: as a balancer in front of Apache or for SSL offloading, it could be needed to increase the tune.bufsize so that it can accept larger requests and larger Cookie sizes.

Haproxy checks the size as (tune.bufsize - tune.maxrewrite) where Maxrewrite is adjusted to half buffersize if that is larger than 1024.

Check the haproxy config at: http://www.haproxy.org/download/1.5/doc/configuration.txt.

Given the above info, tune.bufsize should be set to be double the Apache LimitRequestFieldSize.

Watch out for memory and ensure haproxy has enough resources to work with no issues.

Python based SSH / password client

Previously I have used expect to be able to automating doing ssh to hosts that don't have password-less configured yet.
expect is a bit confusing to use and it took me a bit of time to compile a good reusable expect script. (you can find more info about it in this older post)

Python provides a more elegant solution using the Paramiko SSH2 liberary.
Using this library we can create a simple ssh client that can take a host, a username, a base64 encrypted password and a command string from the command line and then execute it on the remote host.

This can then be used further inside any script similar to the SSH command.

Those scripts are created on Centos 7 using the Python 2.7 Centos packages. Paramiko came along with my Python installation but stil you can install it if it is missing using:

# pip install paramiko

Centos doesn't come with pip, so you need to install it using yum.

Now, let me show a very small password encoder so we can use the base64 encryption to hid the password used on command line:

[root@feanor ~]# cat ./b64encode.py
#!/usr/bin/python

import base64
import sys

string = sys.argv[1]
codedstring = ''
codedstring = base64.b64encode(string)

print string, codedstring
[root@feanor ~]#

[root@feanor ~]# ./b64encode.py mypassword
mypassword bXlwYXNzd29yZA==
[root@feanor ~]#


The above will generate the coded password and can be run in a secure machine, the base64 encoded password then needs to stored in a secured file (user only access , eg: mod 600) and used for our script as below:

[root@feanor ~]# ./sshconnect.py localhost root "`cat ~/passwordfile`" "uname -a;hostname"



Some might argue that this base64 can be cracked easily, but I would say it is better than using a plan text password.



The actual script can be found below:

#!/usr/bin/python

import sys
import paramiko
import base64

if len(sys.argv) < 5:
    print "args missing"
    sys.exit(1)

hostname = sys.argv[1]
username = sys.argv[2]
b64password = sys.argv[3]
command = sys.argv[4]

password = base64.b64decode(b64password)

port = 22
 
try:
    clientSession = paramiko.SSHClient()
    clientSession.load_system_host_keys()
    clientSession.set_missing_host_key_policy(paramiko.WarningPolicy())
    clientSession.connect(hostname, port=port, username=username, password=password)
    stdin, stdout, stderr = clientSession.exec_command(command)
    print stdout.read()
finally:
    clientSession.close()




Again it is always better to use SSH public key authentication, most of the time i use this method once so that i can set the keys and never use the password for logging into a machine any more.
Passwords need to be strong and long 15 chars at least if they must be used.

An example random password generators can be found at:

http://passwordsgenerator.net/  (Should be using https for this site, though it offers a good set of password generation options)
https://www.random.org/passwords/ (Use advanced mode and change the seed)
https://strongpasswordgenerator.com/
https://www.grc.com/passwords.htm   (Really strong 64 char passwords !!)

Changing puppet master used by an agent node

This is how to change the puppet master used by an agent node.
The agent needs to connect to a new master and get its config from that new master onward.

The steps below pertain to Puppet Enterprise agent 3.x.
To do this we basically need to do 3 things.

1- remove the old master SSL certificates from the agent filesystem.
go to the folder: /etc/puppetlabs/puppet under agent filesystem and remove the ssl folder or move it to a new name. then create an empty ssl folder.

2- on the same folder edit the config file puppet.conf
the main section should show the new puppet master hostname in the server tag:

[main]
   server = mynewpuppetmaster.host.name

3- once the above is done, issue a puppet run using:

# puppet agent -t

This will trigger the puppet agent to create a new ssl certs and requests issued for the new puppet master and the cert request will automatically go to the new server.


On the server the request goes to /etc/puppetlabs/puppet/ssl/ca/requests.
you can list the outstanding requests using:

# puppet cert list

from the puppet enterprise web console you can accept the new cert request or you can do from the command line using:

# puppet cert sign mypuppetagent.host.name

One issue i saw while doing this was that the filesystem was 100% full and the cert requests to the master failed and never shown up in the console.
This happened because the request pem file was not able to be written to the requests directory on the master.
The error shown was like this when running:

# puppet cert list
Error: header too long

and also on the client side while running:

# puppet agent -t

Info: Caching certificate for ca
Error: Could not request certificate: Error 400 on SERVER: header too long

To fix this just simply ensure the filesystem used by puppet master is not 100% full.

Hardening Your Apache Configuration

Exposing apache webserver to the public internet is a serious business.
Any exposed service need to be secured and only expose "the needed" functionality "only" as per the application need.

Apache comes by default with many things disabled, still some more work needs to be done to ensure your server is secure and not open to easy attacks.

Below are some examples of configurations that could help make apache more secure.

Hiding the version of Apache:

We need to hid all the info exposed by apache about the host machine and about itself. To do this we need to set the below in the global config section:

ServerSignature Off
ServerTokens Prod

Also we can try to use mod_headers to unset the server header, though most of apache binary distributions has this hard-coded.

Header unset Server

 

Turn off directory browsing:

Apache by default allows directory listings if we use file system derived URLs under the default document root.
This is an issue as it can expose a lot of info about the local files served by apache. to stop this we need to disable indexes in location scopes:

Options -Indexes

Disabling TRACE method:

We need to disable all the HTTP methods that are not going to be used by the application, most importantly Trace method:

To do this we do either of the below configurations:

TraceEnabled Off

or:

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]

Enable Mod_reqtimeout to prevent Slowloris attack:

This is to prevent apache resources from being depleted by slow connections that tries to hold resources for longer duration leading eventually to denial of service.
To battle this attack we need to enable mod_reqtimeout and set its parameters with appropriate values according to the appliaction.
Below values are only for demo perposes:

LoadModule reqtimeout_module    "mod_reqtimeout.so"

<IfModule mod_reqtimeout.c>
   RequestReadTimeout header=120,MinRate=500 body=120,MinRate=500
</IfModule>

Remove the default Error document:

Apache uses a set of default error documents that are presented on http error codes like 404 or 500.
Those pages expose a lot of info about apache and it is better to replace them with custom pages.
A minimum config is shown below for most common error pages:

ErrorDocument 500 "Internal Server Error"
ErrorDocument 404 "The Requested URI is not found "
ErrorDocument 503 "Service Not found"
ErrorDocument 403 "Forbidden"

Disable all not needed modules:

Need to disable all the modules not needed by application, simply comment them and only enable the ones that your application needs.


A more comprehensive list that covers further hardening and security issue fixes can be found in the below link:

https://geekflare.com/apache-web-server-hardening-security/