Using Haproxy to send Proxy-Authorization header to an up stream authenticating proxy

I have been lately seeing a lot of challenge to get certain Java libraries like Eclipse egit to work with https/https proxy with basic authentication.
Java will not accept the http(s) proxy user and password out of the box, code needs to be written to handle those which is not the case in older versions of egit.

To come around this, I have installed HAproxy and used it as an intermediate layer between my scripts and the backend proxy.
To test this I have installed HAproxy 1.6.3 on Linux Mint along with Squid to act as forward proxy with basic authentication enabled.

HAproxy was able to send the Proxy-authorization header for me and hide the complexity of worrying about how to make egit do this.

Below is the HAproxy configuration:

global
        #log /dev/log   local0
        #log /dev/log   local1 notice
        chroot /var/lib/haproxy
        #stats socket /run/haproxy/admin.sock mode 660 level admin
        #stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 1024

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000ms
        timeout client  50000ms
        timeout server  50000ms
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


frontend hideproxy
        bind *:3003
        default_backend authproxy
        option http_proxy
        option http-use-proxy-header

backend authproxy
        server proxyserver localhost:3128
        reqadd Proxy-Authorization:\ Basic\ dXNlcjp1c2Vy

listen stats
    bind        :1936
    stats enable
    stats uri /


To obtain the base64 code for the user and password you can do so by using the below command:

echo -n user:password | openssl enc -a


Once HAproxy is up, the any request to port 3003 will be mapped to the squid proxy on port 3128 with the Proxy-Authorization added to it:

Turin haproxy # curl -I -x http://localhost:3003 https://www.google.com
HTTP/1.1 200 Connection established

HTTP/1.1 200 OK
Date: Thu, 14 Jun 2018 16:10:34 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2018-06-14-16; expires=Sat, 14-Jul-2018 16:10:34 GMT; path=/; domain=.google.com
Set-Cookie: NID=132=kWdwiPFfWHqMQ-X_H9_W08F60_x1eTSIM26K9GEH6TYj--ipq6veJnTM8cZHg9yKYUQWHikVKcBfVg87utujazE3MKhi6q13QoanH_Q8BXaVPpbT8X7URICo4ZlcRvSG; expires=Fri, 14-Dec-2018 16:10:34 GMT; path=/; domain=.google.com; HttpOnly
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="43,42,41,39,35"
Accept-Ranges: none
Vary: Accept-Encoding

Turin haproxy #

How to create a Self-Signed SSL certificate with Subject Alternate Names - SAN

In this post, I want to record the steps to generate a self-signed SSL/TLS certificate with SAN extentions.
This is becoming important as may of the modern browsers now check the SAN section rather than the CN common name for the certificate subject.

To begin, lets generate our root CA certificate:
openssl genrsa -out selfca.key 2048

Then we create the root CA:
[root@localhost ~]# openssl req -x509 -key selfca.key -sha256 -days 2000 -out selfca.crt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:NL    
State or Province Name (full name) []:Noord Holland
Locality Name (eg, city) [Default City]:Amsterdam
Organization Name (eg, company) [Default Company Ltd]:SHERIF Ltd
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []:IT Root CA
Email Address []:
[root@localhost ~]#

Next we create our website private key:
openssl genrsa -out www.sherif.site.key 2048


And then we generate a CSR (certificate signing request):
[root@localhost ~]# openssl req -out www.sherif.site.csr -key www.sherif.site.key -new
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:EG
State or Province Name (full name) []:Cairo
Locality Name (eg, city) [Default City]:Cairo
Organization Name (eg, company) [Default Company Ltd]:SHERIF Ltd
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []:www.sherif.site
Email Address []:root@sherif.site

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@localhost ~]#

Then we sign the certificate and we are done:
[root@localhost ~]# openssl x509 -req -in www.sherif.site.csr -CA selfca.crt -CAkey selfca.key -CAcreateserial -out mydomain.com.crt -days 720 -sha256
Signature ok
subject=/C=EG/ST=Cairo/L=Cairo/O=SHERIF Ltd/OU=IT/CN=www.sherif.site/emailAddress=root@sherif.site
Getting CA Private Key
[root@localhost ~]#


Now lets check the certificate, we can clearly see the information of the CA and also for the certificate owner, but there is no SAN information in the certificate.
[root@localhost ~]# openssl x509 -in www.sherif.site.crt -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            c8:4b:f0:5d:26:4a:18:4b
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=NL, ST=Noord Holland, L=Amsterdam, O=SHERIF Ltd, OU=IT, CN=ROOT CA
        Validity
            Not Before: Jun  2 12:18:17 2018 GMT
            Not After : May 22 12:18:17 2020 GMT
        Subject: C=EG, ST=Cairo, L=Cairo, O=SHERIF Ltd, OU=IT, CN=www.sherif.site/emailAddress=root@sherif.site
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c6:d6:c3:96:6d:0f:b2:63:74:d6:04:7a:2c:c1:
                    e3:bd:6d:6e:0a:89:a4:14:83:de:e6:d9:e9:4b:f1:
                   .....

Adding SAN information:

Now lets try to create a certificate using multiple SAN (Subject alternate names).
To do this, we need to pass the openssl command a customized configuration file while we are generating the CSR.
Below is an example file:

[root@localhost ~]# cat san_config.cfg
[ req ]
default_bits            = 2048
default_md              = sha256
default_keyfile         = privkey.pem
distinguished_name      = req_distinguished_name
x509_extensions = v3_ca # The extensions to add to the self signed cert

prompt = no
string_mask = utf8only

[ req_distinguished_name ]
countryName             = XX
stateOrProvinceName     = State
localityName            = Default City
0.organizationName      = Default Company Ltd
organizationalUnitName  = Unit
commonName              = localhost
emailAddress            = root@localhost

[ v3_ca ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName=@alternate_names

[alternate_names ]
DNS.1        = www
DNS.2        = fingon
DNS.3        = Fingon
[root@localhost ~]#

The file serves 2 purposes, it will automatically fill the CSR question form for us using the values above and will also introduce the SAN extensions to the certificate.
To use this file we add the -config parameter to the openssl command to generate the CSR:

openssl req -out www.sherif.site.csr -key www.sherif.site.key -new -config san_config.cfg

Then we also add the the -extfile & -extensions parameters to the openssl command to generate the certificate:

openssl x509 -req -in www.sherif.site.csr -CA selfca.crt -CAkey selfca.key -CAcreateserial -out www.sherif.site.san.crt -days 720 -sha256 -extfile ./san_config.cfg -extensions 'v3_ca'

Once the Certificate is created, we can check the alternate names section we configured:

[root@localhost ~]# openssl x509 -in www.sherif.site.san.crt -text -noout|grep -i -A3 -B6 fin
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:www, DNS:fingon, DNS:Fingon
    Signature Algorithm: sha256WithRSAEncryption
         a5:e2:a5:89:76:f4:59:ed:46:46:a7:6e:76:ab:ab:ee:e2:cd:
         ef:c9:c4:22:f8:93:92:26:f3:87:51:cd:e7:e7:b5:4e:73:fb:
[root@localhost ~]#
This ensures that all subject alternative names are added to the certificate.

Enjoy SSL 👌

Unix Open files and lsof

The linux lsof when run with no options lists all open files used by all linux processes running on the linux system.
As per the lsof man page the open file can be:
"An  open  file  may be a regular file, a directory, a block special file, a character special file, an executing text reference, a library, a stream or a network file (Internet socket, NFS file or UNIX domain socket.)  A specific file or all the files in a file system may be selected by path.
"
Thus, the output of lsof would almost always be a super set of the process open file descriptors that are limited by linux kernel open files limit.

Lets take the below example for the running terminal emulator:

sherif@fingolfin:~$ ps -ef |grep terminal|grep -v grep
sherif   17612     1  0 18:45 ?        00:00:08 /usr/bin/xfce4-terminal
sherif@fingolfin:~$ lsof -p 17612 |wc -l
182
sherif@fingolfin:~$ ls -l /proc/17612/fd/* |wc -l
16
sherif@fingolfin:~$ lsof -p 17612 |egrep " [0-9]+[rwu] "
xfce4-ter 17612 sherif    0r      CHR                1,3      0t0       6 /dev/null
xfce4-ter 17612 sherif    1w      CHR                1,3      0t0       6 /dev/null
xfce4-ter 17612 sherif    2w      REG                8,1     7452 3805497 /home/sherif/.xsession-errors
xfce4-ter 17612 sherif    3u  a_inode               0,13        0    9863 [eventfd]
xfce4-ter 17612 sherif    4u  a_inode               0,13        0    9863 [eventfd]
xfce4-ter 17612 sherif    5u     unix 0x0000000000000000      0t0   59770 type=STREAM
xfce4-ter 17612 sherif    6u  a_inode               0,13        0    9863 [eventfd]
xfce4-ter 17612 sherif    7u     unix 0x0000000000000000      0t0   59772 type=STREAM
xfce4-ter 17612 sherif    8u  a_inode               0,13        0    9863 [eventfd]
xfce4-ter 17612 sherif    9u     unix 0x0000000000000000      0t0   59773 type=STREAM
xfce4-ter 17612 sherif   10r  a_inode               0,13        0    9863 inotify
xfce4-ter 17612 sherif   11u     unix 0x0000000000000000      0t0   59774 type=STREAM
xfce4-ter 17612 sherif   12u      CHR                5,2      0t0      88 /dev/ptmx
xfce4-ter 17612 sherif   13u      REG                8,1        0 2228246 /tmp/#2228246 (deleted)
xfce4-ter 17612 sherif   14u      REG                8,1        0 2241053 /tmp/#2241053 (deleted)
xfce4-ter 17612 sherif   15u      REG                8,1        0 2241055 /tmp/#2241055 (deleted)
sherif@fingolfin:~$ ls -l /proc/17612/fd/*
lr-x------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/0 -> /dev/null
l-wx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/1 -> /dev/null
lr-x------ 1 sherif sherif 64 mrt 29 18:50 /proc/17612/fd/10 -> anon_inode:inotify
lrwx------ 1 sherif sherif 64 mrt 29 18:50 /proc/17612/fd/11 -> 'socket:[59774]'
lrwx------ 1 sherif sherif 64 mrt 29 18:50 /proc/17612/fd/12 -> /dev/ptmx
lrwx------ 1 sherif sherif 64 mrt 29 18:51 /proc/17612/fd/13 -> '/tmp/#2228246 (deleted)'
lrwx------ 1 sherif sherif 64 mrt 29 18:51 /proc/17612/fd/14 -> '/tmp/#2241053 (deleted)'
lrwx------ 1 sherif sherif 64 mrt 29 18:53 /proc/17612/fd/15 -> '/tmp/#2241055 (deleted)'
l-wx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/2 -> /home/sherif/.xsession-errors
lrwx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/3 -> 'anon_inode:[eventfd]'
lrwx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/4 -> 'anon_inode:[eventfd]'
lrwx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/5 -> 'socket:[59770]'
lrwx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/6 -> 'anon_inode:[eventfd]'
lrwx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/7 -> 'socket:[59772]'
lrwx------ 1 sherif sherif 64 mrt 29 18:45 /proc/17612/fd/8 -> 'anon_inode:[eventfd]'
lrwx------ 1 sherif sherif 64 mrt 29 18:50 /proc/17612/fd/9 -> 'socket:[59773]'
sherif@fingolfin:~$

As you can see, the output of un-customized lsof forms a super set of the actual file descriptors that fall under the Linux openfile limit.

For more information about lsof and how to tweak the output, please check: https://www.thegeekstuff.com/2012/08/lsof-command-examples

Reporting Java system properties default values

In many cases we might want to check what are the default value for a certain Java property.
This helps in tuning Java applications and also in predicting application behavior when based on those default values.

Java sets many properties by default unless you chose to provide other values.
To report all Java properties for a given JVM we use the below form:

java -XX:+PrintFlagsFinal -version

This will list all the applicable properties to that JVM.
To find a certain property on Windows, we can use the findstr command which works like grep:

C:\Users\sherif>java -XX:+PrintFlagsFinal -version|findstr /I heapsize
    uintx ErgoHeapSizeLimit                         = 0
          {product}
    uintx HeapSizePerGCThread                       = 87241520
          {product}
    uintx InitialHeapSize                          := 100663296
          {product}
    uintx LargePageHeapSizeThreshold                = 134217728
          {product}
    uintx MaxHeapSize                              := 1591738368
          {product}
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

C:\Users\sherif>

This shows that the default max heap size is set to be 1.5GB by default for the 64-bit JVM if we didn't specify a heap size.

Another useful option for Java is the showSettings:properties option, this will show system related settings like the OS type, CPU architecture, and boot class path.
The form looks like this:

java -XshowSettings:properties -version

The output is printed on Windows standard error thus normal piping wouldn't work with it.

How to grep for utf8 characters?

UTF8 offers a lot of flexibility to represent literals and shapes (emoicons) that come from different languages other than Latin.
You can check the page: http://www.utf8-chartable.de/unicode-utf8-table.pl which offers a table form for UTF8.

Lets say you want to grep for all lines that contain Arabic characters in a file and you don't know Arabic and can't even read it.
An easy way would be to use Arabic character UTF8 representation if grep is able to understand it.

grep does offer this facility using the -P option.
grep man:
-P, --perl-regexp
              Interpret  PATTERN  as a Perl regular expression.  This is highly experimental and grep -P may warn of unimplemented
              features.

But a better approach is to use the very useful printf command.
printf is meant to replace the good old echo.
printf offers very strong set of escaping and formatting features much superior to any thing echo can offer.

Below is how the you can grep of the arabic character range:

[sherif@thingol ~]$ LANG=c grep [`printf "\xd8\x80"`-`printf "\xdb\xbf"`] mixed_text |cat
س
ي
ا
ر
ا
ت
م
م
ي
ز
ة
ف
ي
س
و
ق
ل
م
س
تع
م
ل
[sherif@thingol ~]$ LANG=c grep -v [`printf "\xd8\x80"`-`printf "\xdb\xbf"`] mixed_text |cat
A
GH
a
rt

Test
[sherif@thingol ~]$


Using the LANG environment variable ensures we are able to display arabic characters.
We use the printf command to represent the arabic characters using their hex values from the unicode table.
We could also use the unicode points using:

[sherif@thingol ~]$ printf "\u0644 \u0628 \n"
ل ب
[sherif@thingol ~]$

This allows to do any kind of string manipulation for unicode characters.

Enjoy playing with unicode. 

Basic Java HotSpot Thread Dump State Analyser

I have been doing a lot of thread dump analysis in the last few months and have been trying to find a tool that would help me do my analysis quickly and precisely.
There is a lot of tools that could be used to do the job for you among those are:
* TDA: https://github.com/irockel/tda  & https://github.com/mkbrv/tda
* IBM JCA: https://www.ibm.com/developerworks/community/groups/service/html/communityview
* Samurai: https://github.com/yusuke/samurai

My favorite tool is Samurai, it offers a very nice view of thread states, with very obvious color codes and offers very reliable lock detection.
Samurai also allows you to follow locked objects from a thread waiting on a lock to the locking thread.

Unfortunately Samurai is not being maintained any more, although the code is available opensource in the above Github repo.
Samurai although still very useful, but it is slow, and at times it gets quite challenging to analyze large dumps with several hundred threads.
Also Samurai only color codes the threads bases on the Thread State class state, while the other tools rely more on the thread status reported based on thread execution and ignores the Thread State class.
This makes analysis more confusing and contradicting at times specially some tools like IBM JCA is not even reporting the State altogether and only relies on execution status.

I wanted to build a tool that allows me more control on what I wanted to see and still be simple to maintain on my side and is quick enough to analyze big dumps.

I started to develop 3 scripts that would help me create a table format for thread states from a set of thread dumps.
The first script would extract the dumps from a log file and write each dump in a single file, then a second awk script parses the thread dump to a thread name, a native id and a status that is a combination of both execution and thread state.
The 3rd script then translate this output to an HTML table.

The code is available on github: https://github.com/sherif-abdelfattah/Thread_dump_analyser

As of now (April 1st 2018), the current out put of the scripts looks like this:

The scripts will run only on Linux, they are mainly using bash and awk.
The scripts are designed to parse Java hotspot thread dumps created for Oracle Java.
I am planning to add more features and enhancements on those scripts once I have a chance to do more coding.

Hope that those scripts could be found useful.

How to check Client SSL cert from certain IPs only

Lately, I had a requirement to configure Apache reverse proxy in such a way to ensure any user logging through the Apache reverse proxy is presenting a valid client certificate if they are not internal users.

Internal users using local network IPs and local domain names should be allowed to access without any certificate validation.

To achieve this, we can use the below simple construct if we are using Apache 2.4.x:

<If "%{REMOTE_ADDR} !~ /^127.0.0.1$/ && %{REMOTE_ADDR} !~ /^192.168.[0-9]+.[0-9]+$/">

        SSLVerifyClient require

        SSLVerifyDepth 2

</If>

In case we are using Apache 2.2, the config needs to involve mod_rewrite as below:

SSLVerifyClient optional

SSLVerifyDepth  2

<Location / >

        Order deny,allow

        Deny from all

        Satisfy any

        Allow from ALL

        RewriteEngine on

        RewriteCond %{SSL:SSL_CLIENT_VERIFY} !^SUCCESS$

        RewriteCond %{REMOTE_ADDR} !^127.0.0.1$

        RewriteCond %{REMOTE_ADDR} !^192.168.1.[1-9]+$

        RewriteRule   ^  -  [F]

</Location>

Using SSL verifyclient optional will still try to verify clients but will not block their access.

This, allows us to test if the verification worked or not with mod_rewrite as seen above.