Apache: How to conditionally inject Authorization header

 In one of the cases I was helping with, I ran into a requirement that needed to have a webhook be sent from Jira to Jenkins to trigger a build based on certain condition.
You can trigger a webhook from Jira based on a Jira search query or on a certain action, eg: upon creation of an issue in a project for example.

The problem is that Jira doesn't support using userinfo fields in the URL, which is actually deprecated and should be used as per the RFC https://tools.ietf.org/html/rfc3986, and thus, Jira would have a way to send authentication information out of the box to Jenkins to trigger a build.

To solve this, I proposed to run Jenkins behind an Apache reverse proxy and use a virtual host definition similar to the below:

<VirtualHost *:80>
     ServerName feanor
     DocumentRoot /var/www/html/

     SetEnvIf Remote_Addr "10.0.0.12" buildtriggerjira
     RequestHeader set Authorization "Basic c2hlcmlmOnNoZXJpZg=="   env=buildtriggerjira

     ProxyPreserveHost on
     ProxyVia on
     ProxyPass "/jenkins" "http://feanor:8080/jenkins"
     ProxyPassReverse "/jenkins" "http://feanor:8080/jenkins"
</VirtualHost>

The above configuration will inject an Authorization header in the incoming request on condition that the remote address is the IP address of the Jira server.
This works by setting an Apache environment variable to a certain value in the SetEnvIf condition and then setting the Authorization header if the variable has that value set.

More checks can be made to harden the condition by checking more request fields like the user-agent string or other headers set by the Jira http client sending the webhook request.

Parallel execution in shell script using GNU parallel

Occasionally, one needs to look for certain strings inside very large files.
Very large file here is a text file that is more than 1GB in size.
The file could be opened by Linux tools such as less, but it becomes challenging to navigate the file and search inside it.
Also if we have a much bigger file, things can become more complex to handle with a single threaded application like less.
Things can become even more challenging if we have multiple files of that size, then one needs to either use grep and sed to look for patterns and do edits without the need to load the file(s) in memory.
Even with those command line filters, which mainly act on the input stream, doing a casual grep can take time to provide results.

To solve this, we would need to use the GNU parallel command to make use of CPU parallel execution.
parallel allows commands to be run in separate processes either on the same host or on multiple hosts, that can run in parallel processes, those would then do processing for parts of the input. parallel then collects the output of those processes and generates the result of the execution jobs.

A simple example for running parallel is below:

sherif@ulmo ~/logs $ < catalina.out parallel --pipe grep -C 4 'OutOfMemory'

The above command pipes the catalina.out that is redirected to standard input to the grep -C to look for the string.
The above form uses defaults block size (1MB) and the default number of jobs will be created (same as number of CPU cores available corresponding to 100%).

Using more optimization for the parallel command, we can achieve more efficiency compared to running a single thread filter, below is a comparison done for execution time for a standard egrep done on a 2GB file and same using parallel with 8 jobs and 100MB block size: 

[root@feanor ~]# time egrep -ni "^/ditto" all_files.txt |wc -l
82

real    0m5.401s
user    0m4.773s
sys     0m0.210s


[root@feanor ~]# time parallel --pipepart --joblog joblog1 -a all_files.txt --block 100M -P8 -k egrep -ni "^/ditto" |wc -l
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:

  O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
  ;login: The USENIX Magazine, February 2011:42-47.

This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.

To silence the citation notice: run 'parallel --bibtex'.

82

real    0m2.819s
user    0m7.224s
sys     0m1.801s
[root@feanor ~]#

As you can see, parallel was able to produce the same result in half the time taken by a single threaded egrep.
The fact that the user time is much higher for execution done by parallel, means that parallel made more use of CPU cores as the total CPU user time is higher than actual time (real) take by the process.
(for more information about the time command check this stackoverflow thread: https://unix.stackexchange.com/questions/40694/why-real-time-can-be-lower-than-user-time)

For more information and examples for using parallel, please check the excellent GNU parallel man page: https://www.gnu.org/software/parallel/man.html

Usign Java Keytool to create a selfsigned SSL Certificate with SAN extentions

Recently I was doing some testing on Java Identity and Access management on premise tools and SAML integration with Apache mod-auth-mellon and other Java applications supporting SAML 2.0.
I needed to create a selfsigned SSL cert for a Java application so that I can ensure the whole SAML traffic is encrypted and secure.
Since I wanted to test multiple apps using multiple hostnames on the same server, I needed to have a generic certificate to save time.
Since I am also not a big fan of wildcard certs, and those also have the limitation of working only under a single top-level domain, I elected to use the Subject Alternative Name SSL certificate extension (SAN); so that I can define multiple definite none wildcard names and I can also have multiple domains and IP addresses if needed.

Creating a SAN SSL cert using keytool turned out to be straight forward, we just need to add the keytool option "-ext" and use the SAN name argument and then list all the SAN types needed.
In below example we use the DNS type and the IP type:

sherif@Luthien:~$ keytool -genkey -keyalg RSA -alias selfsigned -keystore keystore.jks -storepass password -validity 730 -keysize 2048 -ext san=dns:localhost,dns:localhost.com,dns:luthien,dns:redash.luthien.tst,dns:luthein.tst,ip:192.168.56.101

 The resulting certificate would look like below:

sherif@Luthien:~$ keytool -list -v -keystore keystore.jks
Enter keystore password: 
Keystore type: jks
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: selfsigned
Creation date: Feb 8, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=redash.luthien.tst, OU=Sherif Redash test, O=Sherif Tests, L=Cairo, ST=Cairo, C=EG
Issuer: CN=redash.luthien.tst, OU=Sherif Redash test, O=Sherif Tests, L=Cairo, ST=Cairo, C=EG
Serial number: 552e3728
Valid from: Sat Feb 08 14:48:35 CET 2020 until: Mon Feb 07 14:48:35 CET 2022
Certificate fingerprints:
     MD5:  C9:B6:DF:E7:3A:5E:EB:6C:97:C7:CD:8F:99:5A:9A:CD
     SHA1: EE:B7:C6:6F:C5:9C:15:36:5F:A7:12:95:4E:AA:8B:59:FB:08:B6:17
     SHA256: 25:3A:07:8D:FE:5B:FF:74:BE:E7:5F:EA:51:5E:8A:D8:31:41:7E:39:33:36:95:F0:27:C1:D8:2B:CE:54:FB:DA
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
  DNSName: localhost
  DNSName: localhost.com
  DNSName: luthien
  DNSName: redash.luthien.tst
  DNSName: luthein.tst
  IPAddress: 192.168.56.101
]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: F5 18 B2 15 5F 3F B9 63   E3 0F D3 8C 58 5A A0 67  ...._?.c....XZ.g
0010: C7 84 EF E1                                        ....
]
]

*******************************************
******************************************

As you can see, we have created the certificate with the SAN extensions that would allow us to use multiple names.

Please take a look at the Java keytool official documentation from Oracle: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html

Regular experession matching in Postgres

Lately I ran across the need to extract user id text strings from a text colum in a database.
The text could be large and might contain multiple user id matches in the same row.

Thus I imported to the data into Postgres and started looking for Postgres functions that would help me use regular expressions to extract a text part of the column that matches the user id text.

I found a very useful Postgres function: REGEXP_MATCHES('input string', 'regexp pattern', 'flags').
This function works very similar to the sed Unix command, it applies the pattern against input string and allows the use of flag modifiers similar to the one used with sed.

Below is an example of a sample data:

The ids as you can see can be found in any part of the comment-text field.
A query using the  REGEXP_MATCHES function would help extract those Ids into a string arrary type defined in Postgres, the query looks like below:

select REGEXP_MATCHES("comment-text", '\[uidkey\:user[a-fA-F0-9]+\]','g') from comments where "comment-text" like '%[uidkey:%]%';

The output of the query looks like below:

As you can see the REGEXP_MATCHES function matched all 5 ids from the 2 rows that contain them and returned 5 array objects with 1 element each containing those ids.
To cast the array objects to standard text strings we use another Postgres function: ARRAY_TO_STRING(Array_object, 'array element separator', 'optional null element replacement'.

The final version of the query looks like below:

select array_to_string(REGEXP_MATCHES("comment-text", '\[uidkey\:user[a-fA-F0-9]+\]','g'),',')
from comments where "comment-text" like '%[uidkey:%]%';

The output of the query looks like below:

Thus, using those methods, we can extract text from text fields and build more complex queries based on that data.

Enjoy playing with your data !!

Setting RequestHeader in Apache

Apache can be used to inject a Request Header in the incoming request that can be either consumed by Apache or forwarded further to another underlying service, in this case Apache works essentially as a reverse proxy.

In a test setup where Apache works as a reverse proxy in front of tomcat, the below Apache configuration is used to implement the reverse proxy functionality and add a Request Header:

<VirtualHost *>
   <Location "/sherif">
      ProxyPass http://127.0.0.1:8080/sherif
      ProxyPassReverse http://127.0.0.1:8080/sherif
      RequestHeader set myh "valueofarequestheader"
   </Location>
</VirtualHost>

In this setup, tomcat is using default port 8080, it has a defined context path for a dummy application defined in tomcat under its context.xml:

<Context>
    <WatchedResource>WEB-INF/web.xml</WatchedResource>
    <WatchedResource>WEB-INF/tomcat-web.xml</WatchedResource>
    <WatchedResource>${catalina.base}/conf/web.xml</WatchedResource>

    <Manager pathname="/sherif" />
</Context>


Under tomcat webapps, we have folder created as sherif and has a simple index.html file to support the test:
[root@localhost conf]# ls -lt ../webapps/sherif/
total 4
-rw-r--r-- 1 root root 14 Mar 19 16:27 index.html
[root@localhost conf]#

To verify the header being added, we configure tomcat to log the head myh.
This is done on tomcat server.xml accesslog value as below:

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b %{myh}i" />

Once a request is sent to http://localhost/sherif, tomcat logs the below log showing the request header being added by Apache and reaching tomcat:

127.0.0.1 - - [19/Mar/2019:16:46:47 -0400] "GET /sherif/ HTTP/1.1" 200 17 valueofarequestheader

This configuration is useful in passing headers to backend services in case those are not already sent by the source user agent.

Using Haproxy to send Proxy-Authorization header to an up stream authenticating proxy

I have been lately seeing a lot of challenge to get certain Java libraries like Eclipse egit to work with https/https proxy with basic authentication.
Java will not accept the http(s) proxy user and password out of the box, code needs to be written to handle those which is not the case in older versions of egit.

To come around this, I have installed HAproxy and used it as an intermediate layer between my scripts and the backend proxy.
To test this I have installed HAproxy 1.6.3 on Linux Mint along with Squid to act as forward proxy with basic authentication enabled.

HAproxy was able to send the Proxy-authorization header for me and hide the complexity of worrying about how to make egit do this.

Below is the HAproxy configuration:

global
        #log /dev/log   local0
        #log /dev/log   local1 notice
        chroot /var/lib/haproxy
        #stats socket /run/haproxy/admin.sock mode 660 level admin
        #stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 1024

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000ms
        timeout client  50000ms
        timeout server  50000ms
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


frontend hideproxy
        bind *:3003
        default_backend authproxy
        option http_proxy
        option http-use-proxy-header

backend authproxy
        server proxyserver localhost:3128
        reqadd Proxy-Authorization:\ Basic\ dXNlcjp1c2Vy

listen stats
    bind        :1936
    stats enable
    stats uri /


To obtain the base64 code for the user and password you can do so by using the below command:

echo -n user:password | openssl enc -a


Once HAproxy is up, the any request to port 3003 will be mapped to the squid proxy on port 3128 with the Proxy-Authorization added to it:

Turin haproxy # curl -I -x http://localhost:3003 https://www.google.com
HTTP/1.1 200 Connection established

HTTP/1.1 200 OK
Date: Thu, 14 Jun 2018 16:10:34 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2018-06-14-16; expires=Sat, 14-Jul-2018 16:10:34 GMT; path=/; domain=.google.com
Set-Cookie: NID=132=kWdwiPFfWHqMQ-X_H9_W08F60_x1eTSIM26K9GEH6TYj--ipq6veJnTM8cZHg9yKYUQWHikVKcBfVg87utujazE3MKhi6q13QoanH_Q8BXaVPpbT8X7URICo4ZlcRvSG; expires=Fri, 14-Dec-2018 16:10:34 GMT; path=/; domain=.google.com; HttpOnly
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="43,42,41,39,35"
Accept-Ranges: none
Vary: Accept-Encoding

Turin haproxy #

How to create a Self-Signed SSL certificate with Subject Alternate Names - SAN

In this post, I want to record the steps to generate a self-signed SSL/TLS certificate with SAN extentions.
This is becoming important as may of the modern browsers now check the SAN section rather than the CN common name for the certificate subject.

To begin, lets generate our root CA certificate:
openssl genrsa -out selfca.key 2048

Then we create the root CA:
[root@localhost ~]# openssl req -x509 -key selfca.key -sha256 -days 2000 -out selfca.crt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:NL    
State or Province Name (full name) []:Noord Holland
Locality Name (eg, city) [Default City]:Amsterdam
Organization Name (eg, company) [Default Company Ltd]:SHERIF Ltd
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []:IT Root CA
Email Address []:
[root@localhost ~]#

Next we create our website private key:
openssl genrsa -out www.sherif.site.key 2048


And then we generate a CSR (certificate signing request):
[root@localhost ~]# openssl req -out www.sherif.site.csr -key www.sherif.site.key -new
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:EG
State or Province Name (full name) []:Cairo
Locality Name (eg, city) [Default City]:Cairo
Organization Name (eg, company) [Default Company Ltd]:SHERIF Ltd
Organizational Unit Name (eg, section) []:IT
Common Name (eg, your name or your server's hostname) []:www.sherif.site
Email Address []:root@sherif.site

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@localhost ~]#

Then we sign the certificate and we are done:
[root@localhost ~]# openssl x509 -req -in www.sherif.site.csr -CA selfca.crt -CAkey selfca.key -CAcreateserial -out mydomain.com.crt -days 720 -sha256
Signature ok
subject=/C=EG/ST=Cairo/L=Cairo/O=SHERIF Ltd/OU=IT/CN=www.sherif.site/emailAddress=root@sherif.site
Getting CA Private Key
[root@localhost ~]#


Now lets check the certificate, we can clearly see the information of the CA and also for the certificate owner, but there is no SAN information in the certificate.
[root@localhost ~]# openssl x509 -in www.sherif.site.crt -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            c8:4b:f0:5d:26:4a:18:4b
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=NL, ST=Noord Holland, L=Amsterdam, O=SHERIF Ltd, OU=IT, CN=ROOT CA
        Validity
            Not Before: Jun  2 12:18:17 2018 GMT
            Not After : May 22 12:18:17 2020 GMT
        Subject: C=EG, ST=Cairo, L=Cairo, O=SHERIF Ltd, OU=IT, CN=www.sherif.site/emailAddress=root@sherif.site
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c6:d6:c3:96:6d:0f:b2:63:74:d6:04:7a:2c:c1:
                    e3:bd:6d:6e:0a:89:a4:14:83:de:e6:d9:e9:4b:f1:
                   .....

Adding SAN information:

Now lets try to create a certificate using multiple SAN (Subject alternate names).
To do this, we need to pass the openssl command a customized configuration file while we are generating the CSR.
Below is an example file:

[root@localhost ~]# cat san_config.cfg
[ req ]
default_bits            = 2048
default_md              = sha256
default_keyfile         = privkey.pem
distinguished_name      = req_distinguished_name
x509_extensions = v3_ca # The extensions to add to the self signed cert

prompt = no
string_mask = utf8only

[ req_distinguished_name ]
countryName             = XX
stateOrProvinceName     = State
localityName            = Default City
0.organizationName      = Default Company Ltd
organizationalUnitName  = Unit
commonName              = localhost
emailAddress            = root@localhost

[ v3_ca ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName=@alternate_names

[alternate_names ]
DNS.1        = www
DNS.2        = fingon
DNS.3        = Fingon
[root@localhost ~]#

The file serves 2 purposes, it will automatically fill the CSR question form for us using the values above and will also introduce the SAN extensions to the certificate.
To use this file we add the -config parameter to the openssl command to generate the CSR:

openssl req -out www.sherif.site.csr -key www.sherif.site.key -new -config san_config.cfg

Then we also add the the -extfile & -extensions parameters to the openssl command to generate the certificate:

openssl x509 -req -in www.sherif.site.csr -CA selfca.crt -CAkey selfca.key -CAcreateserial -out www.sherif.site.san.crt -days 720 -sha256 -extfile ./san_config.cfg -extensions 'v3_ca'

Once the Certificate is created, we can check the alternate names section we configured:

[root@localhost ~]# openssl x509 -in www.sherif.site.san.crt -text -noout|grep -i -A3 -B6 fin
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:www, DNS:fingon, DNS:Fingon
    Signature Algorithm: sha256WithRSAEncryption
         a5:e2:a5:89:76:f4:59:ed:46:46:a7:6e:76:ab:ab:ee:e2:cd:
         ef:c9:c4:22:f8:93:92:26:f3:87:51:cd:e7:e7:b5:4e:73:fb:
[root@localhost ~]#
This ensures that all subject alternative names are added to the certificate.

Enjoy SSL 👌