Thursday 22 December 2016

Rundeck Parametric Job Definition

I have a use case where I need to run same set of shell instructions multiple times with different users and compare the results on remote hosts.

Thought about automating this using Rundeck, and since I am not allowed to use ssh keys password-less authentication in this case, Rundeck would be ideal.

below is a sample job listing (XML) for a parametric job that accepts username and a password and just prints the username in a simple embedded shell script.

<joblist>
  <job>
    <context>
      <options preserveOrder='true'>
        <option name='Password' required='true' secure='true' valueExposed='true' />
        <option name='User_Name' required='true' value='App_User' />
      </options>
    </context>
    <description></description>
    <executionEnabled>true</executionEnabled>
    <id>5e63d0fa-9bae-4d1d-bc09-a2d8429f8b19</id>
    <loglevel>INFO</loglevel>
    <name>Parametrized_job1</name>
    <scheduleEnabled>true</scheduleEnabled>
    <sequence keepgoing='false' strategy='node-first'>
      <command>
        <script><![CDATA[echo "This is a test Job"
echo 'using form $RD_OPTION_USER_NAME' $RD_OPTION_USER_NAME
echo 'using form ${option.User_name}' ${option.User_Name} #doesn't work :)
echo 'using form \@option.User_name\@' @option.User_Name@
]]></script>
        <scriptargs />
      </command>
    </sequence>
    <uuid>5e63d0fa-9bae-4d1d-bc09-a2d8429f8b19</uuid>
  </job>
</joblist>


The job parameters are better accessed as passed by Rundeck env. as:$RD_OPTION_paramname

The above job option env. variable will be passed to all rundeck executed scripts so it is much easier to use it in this form than the one between @ signs.
The form ${option.User_Name} is not working in this case as Rundeck will use it as a command-line argument if we are running a commad step rather than an embedded script.

below is an example job definition:

<joblist>
  <job>
    <context>
      <options preserveOrder='true'>
        <option name='Password' required='true' secure='true' valueExposed='true' />
        <option name='User_Name' required='true' value='App_User' />
      </options>
    </context>
    <description></description>
    <executionEnabled>true</executionEnabled>
    <id>5d1db9a0-d381-49ec-8981-c483b375751c</id>
    <loglevel>INFO</loglevel>
    <name>Parametrized_job2</name>
    <scheduleEnabled>true</scheduleEnabled>
    <sequence keepgoing='false' strategy='node-first'>
      <command>
        <exec>echo ${option.User_Name}</exec>
      </command>
    </sequence>
    <uuid>5d1db9a0-d381-49ec-8981-c483b375751c</uuid>
  </job>
</joblist>


Rundeck parametric jobs will prove very useful for doing generic tasks and generic automation.

Please check more info in Rundeck documentation at: http://rundeck.org/1.6.2/manual/job-options.html#prompting-the-user



Monday 5 December 2016

JQuery Draggable and Droppable demo

I was exploring a way to use simple HTML UI components to offer UI drag and drop functionality.
This is a long term plan I was having to start real automation of my day to day work and avoid executing or editing shell scripts or puppet manifests.

JQuery offers a good simple API to allow drag and drop fuctionaltity with a lot of flexiability.

Since I am not UX-UI designer, it took me quite sometime to understand how it works and write down a working demo code for the functionality.

Below is a simple code for simple HTML table (TD elements) having draggable numbers from 1 to 8 and are only droppable inside the HTML table.

<html>

<head>
<style>
table, th, td {
    border: 5px solid black;
    text-align: center;
    color: blue;
    font-size: 300%;
    font-weight: bold;
    border-spacing: 10px;
}

</style>


<script src="https://code.jquery.com/jquery-1.12.4.js"></script>
<script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>

<script>

// Make the element with id "draggable" draggable
$(function () {
   
    count = 0;

    $("#td1").draggable({revert: "invalid", containment: "document", grid: [220,210] });
    $("#td2").draggable({revert: "invalid", containment: "document", grid: [220,210] });
    $("#td3").draggable({revert: "invalid", containment: "document", grid: [220,210]});
    $("#td4").draggable({revert: "invalid", containment: "document", grid: [220,210]});
    $("#td5").draggable({revert: "invalid", containment: "document", grid: [220,210]});
    $("#td6").draggable({revert: "invalid", containment: "document", grid: [220,210]});
    $("#td7").draggable({revert: "invalid", containment: "document", grid: [220,210]});
    $("#td8").draggable({revert: "invalid", containment: "document", grid: [220,210]});
   
    $( "#tb1" ).droppable({
      drop: function(event, ui) {
      Edit_dragable (  ui.draggable );
      //$("#logger").html( "Dropped! ");
      //$("#logger2").html( "Testing" );
    }
    });
   
   
        function Edit_dragable( $item ) {
        $item
          .css( "color", "Yellow" );
          var textval = 'The number is ' + $item.html() + ' & the td id was: ' + $item.attr('id');
          $("#logger2").html( textval );
      }
   
});
</script>

</head>

<table id="tb1" >
<caption>Test arrange game </caption>
 <tr id="tr1">
    <td id="td1" style="width:200;  height:200; border-color: green"> 1 </td>
    <td id="td2" style="width:200;  height:200;"> 2 </td>
    <td id="td3" style="width:200;  height:200;"> 3 </td>
</tr>

 <tr id="tr2">
    <td id="td4" style="width:200;  height:200"> 4 </td>
    <td id="td5" style="width:200;  height:200;"> 5 </td>
    <td id="td6" style="width:200;  height:200;"> 6 </td>
</tr>

 <tr id="tr3">
    <td id="td7" style="width:200;  height:200;"> 7 </td>
    <td id="td8" style="width:200;  height:200;"> 8 </td>
   
    <!--<td id="td9" draggable="true" ondragstart="drag(event)"> 9 </td>-->
</tr>
</table>

<p id="logger"> </p>
<p id="logger2"> </p>


The code involves some HTML, CSS styles and JQuery to be able to achieve the functionality.

Next steps is to combine this with my other code components to offer an interactive control dash board for automating my day to day tasks.
Will continue to post as I progress in this work.




Sunday 25 September 2016

Haproxy CSV stats

Haproxy exposes a very useful statistics page that can be accessed over a web browser or from the command line using a tool like curl.
Haproxy allows the stats to be exposed in a CSV format that is exceedingly useful if you are going to have a script around it.

To access the CSV stats from http interface use the below:

curl  -u statsuser:statspassword 'http://haproxyhost:8001/stats/?stats;csv;' 2>/dev/null |egrep -iv "up|^#|open" |cut -d"," -f1,2,18

Where port 8001 is the statistics port as defined in the haproxy config.
The config is simple, already shown in older posts, should be something like this:

listen stats *:8001
    stats enable
    stats uri /stats
    stats hide-version
    stats auth statsuser:statspassword



The above mini script will print out only the apps and backends that show as down.
working ones will not show up.
The CVS header is also stripped as the header always starts with a "#" this makes it possible to process only the data using a grep -v as above.

The CSV result contains a big amount of info that can be used for load management and automation.
The details of the haproxy CSV stats layout can be found at:
https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#9  

Thursday 22 September 2016

Updating a field of a DB table using records from same table

The below query is to update a field that is set to null in a config table from records that contain values in the same table.

I came across this scenario while working on a puppet upgrade project where we are moving the file based Yaml configs to SQL DB.


below is the query:

update config_dev c
set groupname = l.groupname
from ( select distinct (groupname) , name from config_dev where groupname is not null ) l
where c.groupname is null
and c.name = l.name

The table contains some "groupname" fields that contains null, still those should contain same group values as others given that name field is the same.


the above query will do that for us.
Not that the above query will not run in MySQL !!

Thanks to my friend Mohamed Youssef who helped me with this query.



Monday 19 September 2016

Setting up a new PostgreSQL on Redhat systems

Postgres setup turns out to be not as straight forward as I assumed.
When you setup the DB server using yum, the DB comes in a not initialized state, and thus it fails to start using standard service commands.

The below Fedora project wiki link entails a lot of details about this issue:
https://fedoraproject.org/wiki/PostgreSQL

An example on CentOS 7 looks like below.
Once you initialize the DB it starts up with no issues.

[root@vardamir ~]# systemctl start postgresql
Job for postgresql.service failed because the control process exited with error code. See "systemctl status postgresql.service" and "journalctl -xe" for details.
[root@vardamir ~]# journalctl -xe
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A new session with the ID 3 has been created for the user root.
--
-- The leading process of the session is 11388.
Sep 19 11:54:29 vardamir systemd[1]: Starting Session 3 of user root.
-- Subject: Unit session-3.scope has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-3.scope has begun starting up.
Sep 19 11:54:29 vardamir sshd[11388]: pam_unix(sshd:session): session opened for user root by (uid=0)
Sep 19 11:54:30 vardamir dbus[725]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Sep 19 11:54:30 vardamir dbus-daemon[725]: dbus[725]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Sep 19 11:54:30 vardamir dbus[725]: [system] Successfully activated service 'org.freedesktop.problems'
Sep 19 11:54:30 vardamir dbus-daemon[725]: dbus[725]: [system] Successfully activated service 'org.freedesktop.problems'
Sep 19 11:54:52 vardamir polkitd[1394]: Registered Authentication Agent for unix-process:11426:31997 (system bus name :1.28 [/usr/bin/pkttyagent --notify-fd 5 --fallback],
Sep 19 11:54:52 vardamir systemd[1]: Starting PostgreSQL database server...
-- Subject: Unit postgresql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql.service has begun starting up.
Sep 19 11:54:52 vardamir postgresql-check-db-dir[11431]: "/var/lib/pgsql/data" is missing or empty.
Sep 19 11:54:52 vardamir postgresql-check-db-dir[11431]: Use "postgresql-setup initdb" to initialize the database cluster.
Sep 19 11:54:52 vardamir postgresql-check-db-dir[11431]: See /usr/share/doc/postgresql-9.2.15/README.rpm-dist for more information.
Sep 19 11:54:52 vardamir systemd[1]: postgresql.service: control process exited, code=exited status=1
Sep 19 11:54:52 vardamir systemd[1]: Failed to start PostgreSQL database server.
-- Subject: Unit postgresql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql.service has failed.
--
-- The result is failed.
Sep 19 11:54:52 vardamir systemd[1]: Unit postgresql.service entered failed state.
Sep 19 11:54:52 vardamir systemd[1]: postgresql.service failed.
Sep 19 11:54:52 vardamir polkitd[1394]: Unregistered Authentication Agent for unix-process:11426:31997 (system bus name :1.28, object path /org/freedesktop/PolicyKit1/Authe
[root@vardamir ~]# postgresql-setup initdb
Initializing database ... OK

[root@vardamir ~]# systemctl status postgresql
● postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2016-09-19 11:58:57 EDT; 10s ago
  Process: 11542 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
  Process: 11537 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
 Main PID: 11546 (postgres)
   CGroup: /system.slice/postgresql.service
           ├─11546 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
           ├─11547 postgres: logger process  
           ├─11549 postgres: checkpointer process  
           ├─11550 postgres: writer process  
           ├─11551 postgres: wal writer process  
           ├─11552 postgres: autovacuum launcher process  
           └─11553 postgres: stats collector process  

Sep 19 11:58:56 vardamir systemd[1]: Starting PostgreSQL database server...
Sep 19 11:58:57 vardamir systemd[1]: Started PostgreSQL database server.
[root@vardamir ~]#


To allow the postgresql server (postmaster) to listen on all interfaces, we need to pieces of config changes, 1 in postgresql.conf and other in pg_hba.conf as below.
in postgresql.conf we need to set :

listen_addresses = '*' 

so that the postmaster server listens on all interfaces.
second we need to add a host entry in pg_hba.conf so that clients are allowed to connect.
this config will depend on network config of the installed DB, an example is shown below:

host    all             all             10.0.0.0/8              md5

md5 is used to allow encrypted password authentication.
Don't forget to allow the Linux firewall to open port 5432 so that external users can access the newly created DB server.

Its always better to create another user than the default postgres user and use that for the DB work remotely.



Thursday 15 September 2016

Apache / haproxy large Cookie size issues

Although HTTP protocol doesn't have a limit on the cookie size, which is in turn a part of the request/response headers, Apache does impose a limit to protect the webserver from denial of service attaches.

Apache controls this using the "LimitRequestFieldSize" directive which defaults to 8192 byte as a max for a header field.

If larger cookies are needed, we need to bump up this value to a bigger number.
eg:

LimitRequestFieldSize 18192

in order to test this, i set up a simple webserver config with the below config:

RequestHeader unset If-Modified-Since
RequestHeader unset If-None-Match

LimitRequestFieldSize 18192

Header set Set-Cookie "apachenode=node1; path=/;"
Header add Set-Cookie "env=test; path=/;"
Header add Set-Cookie "TimeS=NONE; path=/;"


The LimitRequestFieldSize only works on Requests not on the response, thus apache can set-Cookies with large values with no problems, its the job of the browser to validate this.

To be able to test this from Firefox, I used the Cookie Manager plugin https://addons.mozilla.org/en-US/firefox/addon/cookies-manager-plus/ to set a large cookie that is 24k in size with a random string generated from the site: http://textmechanic.com/text-tools/randomization-tools/random-string-generator/

I managed to prove the compiled limit in Centos 7 is indeed 8192 and expanded that to ~18k as seen above and it worked.

When apache fails to accept the request it responses with an http 400 error as below:


I tried to do the same test with commandline curl but seems curl truncates the large cookie when using the -b option and passing a file as below:

[root@feanor conf]# curl -b ./cookies.txt  http://localhost/new.html
this is a new page :)
[root@feanor conf]# ls -ltrh ./cookies.txt
-rw-r--r-- 1 root root 25K Sep 15 12:28 ./cookies.txt
[root@feanor conf]#


Although the cookie file contains a single cookie that is 24k in size, curl seems to have truncated it and the request went in. 
lowering the LimitRequestFieldSize to something like 4600 managed to have curl reproduce the same behaviors as the browsers:

[root@feanor conf]# curl -b ./cookies.txt  http://localhost/new.html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Size of a request header field exceeds server limit.<br />
<pre>

</pre>
</p>
</body></html>
[root@feanor conf]#

Thus, curl could be tricking you if you are debugging an issue like this from the command line.

Also one more note about large Cookie sizes, if haproxy is used in the setup, eg: as a balancer in front of Apache or for SSL offloading, it could be needed to increase the tune.bufsize so that it can accept larger requests and larger Cookie sizes.

Haproxy checks the size as (tune.bufsize - tune.maxrewrite) where Maxrewrite is adjusted to half buffersize if that is larger than 1024.
Given the above info, tune.bufsize should be set to be double the Apache LimitRequestFieldSize.
Watch out for memory and ensure haproxy has enough resources to work with no issues.



Thursday 25 August 2016

Python based SSH / password client

Previously I have used expect to be able to automating doing ssh to hosts that don't have password-less configured yet.
expect is a bit confusing to use and it took me a bit of time to compile a good reusable expect script. (you can find more info about it in this older post)

Python provides a more elegant solution using the Paramiko SSH2 liberary.
Using this library we can create a simple ssh client that can take a host, a username, a base64 encrypted password and a command string from the command line and then execute it on the remote host.

This can then be used further inside any script similar to the SSH command.

Those scripts are created on Centos 7 using the Python 2.7 Centos packages. Paramiko came along with my Python installation but stil you can install it if it is missing using:

# pip install paramiko

Centos doesn't come with pip, so you need to install it using yum.

Now, let me show a very small password encoder so we can use the base64 encryption to hid the password used on command line:

[root@feanor ~]# cat ./b64encode.py
#!/usr/bin/python

import base64
import sys

string = sys.argv[1]
codedstring = ''
codedstring = base64.b64encode(string)

print string, codedstring
[root@feanor ~]#


[root@feanor ~]# ./b64encode.py mypassword
mypassword bXlwYXNzd29yZA==
[root@feanor ~]#



The above will generate the coded password and can be run in a secure machine, the base64 encoded password then needs to stored in a secured file (user only access , eg: mod 600) and used for our script as below:

[root@feanor ~]# ./sshconnect.py localhost root "`cat ~/passwordfile`" "uname -a;hostname"



Some might argue that this base64 can be cracked easily, but I would say it is better than using a plan text password.



The actual script can be found below:

#!/usr/bin/python

import sys
import paramiko
import base64

if len(sys.argv) < 5:
    print "args missing"
    sys.exit(1)

hostname = sys.argv[1]
username = sys.argv[2]
b64password = sys.argv[3]
command = sys.argv[4]

password = base64.b64decode(b64password)

port = 22
 

try:
    clientSession = paramiko.SSHClient()
    clientSession.load_system_host_keys()
    clientSession.set_missing_host_key_policy(paramiko.WarningPolicy())
    clientSession.connect(hostname, port=port, username=username, password=password)
    stdin, stdout, stderr = clientSession.exec_command(command)
    print stdout.read()
finally:
    clientSession.close()





Again it is always better to use SSH public key authentication, most of the time i use this method once so that i can set the keys and never use the password for logging into a machine any more.
Passwords need to be strong and long 15 chars at least if they must be used.

An example random password generators can be found at:

http://passwordsgenerator.net/  (Should be using https for this site, though it offers a good set of password generation options)
https://www.random.org/passwords/ (Use advanced mode and change the seed)
https://strongpasswordgenerator.com/
https://www.grc.com/passwords.htm   (Really strong 64 char passwords !!)

Wednesday 24 August 2016

Changing puppet master used by an agent node

This is how to change the puppet master used by an agent node.
The agent needs to connect to a new master and get its config from that new master onward.

The steps below pertain to Puppet Enterprise agent 3.x.
To do this we basically need to do 3 things.

1- remove the old master SSL certificates from the agent filesystem.
go to the folder: /etc/puppetlabs/puppet under agent filesystem and remove the ssl folder or move it to a new name. then create an empty ssl folder.

2- on the same folder edit the config file puppet.conf
the main section should show the new puppet master hostname in the server tag:

[main]
   server = mynewpuppetmaster.host.name

3- once the above is done, issue a puppet run using:

# puppet agent -t

This will trigger the puppet agent to create a new ssl certs and requests issued for the new puppet master and the cert request will automatically go to the new server.


On the server the request goes to /etc/puppetlabs/puppet/ssl/ca/requests.
you can list the outstanding requests using:

# puppet cert list

from the puppet enterprise web console you can accept the new cert request or you can do from the command line using:

# puppet cert sign mypuppetagent.host.name

One issue i saw while doing this was that the filesystem was 100% full and the cert requests to the master failed and never shown up in the console.
This happened because the request pem file was not able to be written to the requests directory on the master.
The error shown was like this when running:

# puppet cert list
Error: header too long

and also on the client side while running:

# puppet agent -t

Info: Caching certificate for ca
Error: Could not request certificate: Error 400 on SERVER: header too long


To fix this just simply ensure the filesystem used by puppet master is not 100% full.


Wednesday 17 August 2016

Hardening Your Apache Configuration

Exposing apache webserver to the public internet is a serious business.
Any exposed service need to be secured and only expose "the needed" functionality "only" as per the application need.

Apache comes by default with many things disabled, still some more work needs to be done to ensure your server is secure and not open to easy attacks.

Below are some examples of configurations that could help make apache more secure.

Hiding the version of Apache:

We need to hid all the info exposed by apache about the host machine and about itself. To do this we need to set the below in the global config section:

ServerSignature Off
ServerTokens Prod


Also we can try to use mod_headers to unset the server header, though most of apache binary distributions has this hard-coded.

Header unset Server

 

Turn off directory browsing:

Apache by default allows directory listings if we use file system derived URLs under the default document root.
This is an issue as it can expose a lot of info about the local files served by apache. to stop this we need to disable indexes in location scopes:

Options -Indexes

Disabling TRACE method:

We need to disable all the HTTP methods that are not going to be used by the application, most importantly Trace method:

To do this we do either of the below configurations:

TraceEnabled Off

or:

RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^TRACE
RewriteRule .* - [F]


Enable Mod_reqtimeout to prevent Slowloris attack:


This is to prevent apache resources from being depleted by slow connections that tries to hold resources for longer duration leading eventually to denial of service.
To battle this attack we need to enable mod_reqtimeout and set its parameters with appropriate values according to the appliaction.
Below values are only for demo perposes:

LoadModule reqtimeout_module    "mod_reqtimeout.so"

<IfModule mod_reqtimeout.c>
   RequestReadTimeout header=120,MinRate=500 body=120,MinRate=500
</IfModule>


Remove the default Error document:

Apache uses a set of default error documents that are presented on http error codes like 404 or 500.
Those pages expose a lot of info about apache and it is better to replace them with custom pages.
A minimum config is shown below for most common error pages:

ErrorDocument 500 "Internal Server Error"
ErrorDocument 404 "The Requested URI is not found "
ErrorDocument 503 "Service Not found"
ErrorDocument 403 "Forbidden"

Disable all not needed modules:

Need to disable all the modules not needed by application, simply comment them and only enable the ones that your application needs.


A more comprehensive list that covers further hardening and security issue fixes can be found in the below link:

https://geekflare.com/apache-web-server-hardening-security/

Wednesday 10 August 2016

Overiding backend service error page when using apache proxypass config

Most of the apps that I have supported before are large enterprise grade apps that use stand alone content management software and a large team of content authoring and publishing.
Most of the time those guys take care of every content related staff like site version notes and error pages.

I came across a smaller project this week and I was asked to handle the errors coming out of the application for security reasons.
The application was hosted by tomcat and had apache infront of it to do proxy pass work and Shibboleth authentication.

Tomcat default error page is is very basic yet give a way a lot of info about the tomcat version your application runs, thus it is a good idea to hid it if possible.

Simplist way to do this is from apache using the ProxyErrorOverride set to on as seen below:


ProxyPass "/" "http://backend_tomcat_host:8080/"
ProxyPassReverse "/" "http://backend_tomcat_host:8080/"

ProxyErrorOverride On
ErrorDocument 500 "Internal Server Error"
ErrorDocument 404 "The Requested URI is not found "
ErrorDocument 503 "Service Not found"
ErrorDocument 403 "Forbidden"

The above will offer the user the most basic error pages possible and will hid all the Tomcat details.
More complex Error pages can be used by replacing the above simple text with a URI of an html error page.

For apache 2.2.x the full documentation is available at: http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyerroroverride


Monday 8 August 2016

RabbitMQ Clustering configurations

In this post we will see how to cluster rabbitMQ in 2 configurations.
one using same host, called a vertical cluster, and other using 2 differents hosts called the horizontal cluster.


Create Vertical clustering


1- Start RabbitMQ with differant node names and port numbers similar to below:

RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit ./rabbitmq-server -detached
RABBITMQ_NODE_PORT=5673 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15673}]" RABBITMQ_NODENAME=rabbit2 ./rabbitmq-server -detached

 
using the above env. variable form of starting up rabbitMQ vs using config file; enables us to use diffent ports and nodenames.
The config file should be empty in this case.
The above will start 2 RabbitMQ nodes on the same VM each one has its unique name and has its own uniq set of ports.
2- To enable the 2 nodes to be in a single cluster with one node as DISC and one as RAM do the below:

./rabbitmqctl -n rabbit2@beren stop_app -> stop the rabbitMQ application but node is still running on host beren
./rabbitmqctl -n rabbit2@beren join_cluster --ram rabbit@beren -> Add rabbit2 as RAM node to rabbit cluster, default is DISC
./rabbitmqctl -n rabbit2@beren start_app -> Start the rabbitMQ application
./rabbitmqctl -n rabbit@beren cluster_status -> check the cluster status should show something like this:

Cluster status of node rabbit@beren ...
[{nodes,[{disc,[rabbit@beren]},{ram,[rabbit2@beren]}]},
{running_nodes,[rabbit2@beren,rabbit@beren]},
{partitions,[]}]
...done.

 
3- to change node type from RAM to DISC do:

./rabbitmqctl -n rabbit2@beren stop_app
./rabbitmqctl -n rabbit2@beren change_cluster_node_type disc
./rabbitmqctl -n rabbit2@beren start_app

 
to change it back to RAM do:

./rabbitmqctl -n rabbit2@beren stop_app
./rabbitmqctl -n rabbit2@beren change_cluster_node_type ram
./rabbitmqctl -n rabbit2@beren start_app

Note that at least the cluster should have one disc node. Ram nodes don't offer faster response in queue performance but only offer better speed on queue creation, recommended to have at least 1 disc node on every VM to allow high availability and add ram nodes in a vertical cluster as decscribed above IF needed.

Create Horizontal clustering


To do this we need a second VM running RabbitMQ.
on the new 2nd node do the following.
1- copy the erlang cookie file from the old VM RabbitMQ user home directory to the new VM user home directory.
cookie file name is .erlang.cookie

2- go to RABBITMQ_HOME/sbin & start rabbitMQ

./rabbitmq-server -detached

3- Run the below list of commands:

./rabbitmqctl -n rabbit@vardamir stop_app
./rabbitmqctl -n rabbit@vardamir join_cluster rabbit@beren
./rabbitmqctl -n rabbit@vardamir start_app


All the queues and topics created will automatically be transfered to the new joining node.
This will create a 2 node on 2 VM cluster that will offer true HA for the rabbitMQ. queues can be configured to work in HA mode by creating a policy from the rabbitMQ admin console, the policy should containt ha-mode:all so that the queues are all working across the cluster in HA.
Also replication can be enabled in the same manner.

HAproxy config


HA proxy can be used to offer true load-balancing for a rabbitMQ cluster.
The cluster doesn't offer a common gateway by default so to benefit from it we need balancer sitting infornt of it.
In this case haproxy can help as it is very light weight and a performer.
The config for balancing 2 nodes should look like this:

# Simple configuration for an HTTP proxy listening on port 8080 on all
# interfaces and forwarding requests to a single backend "servers" with a
# single server "server1" listening on 127.0.0.1:8000

    global
    log 127.0.0.1 local0 info
    daemon
   
    defaults
    log     global
    mode     tcp
    option tcplog
    option dontlognull
    retries 3
    option redispatch
    maxconn    2000
    timeout connect 10s
    timeout client     120s
    timeout    server     120s
   
    listen rabbitmq_local_cluster 0.0.0.0:8080
   
    balance roundrobin
    server rabbitnode1    beren:5672 check inter 5000 rise 2 fall 3 #Disk Node
    server rabbitnode2 vardamir:5672 check inter 5000 rise 2 fall 3 #Disk Node
   
   
    listen private_monitoring  :8101
    mode http
    option httplog
    stats enable
    stats uri /stats
    stats refresh  30s


The above makes use of haproxy tcp mode since rabbitMQ uses amqp which is not http compatible.


Tuesday 26 July 2016

Setting up RabbitMQ and its config files

This post is to document how to install rabbitMQ using the generic linux distribution that comes as a tar.gz package.

To proceed with the installation follow the below steps:
1- download latest RabbitMQ from pivotal: https://network.pivotal.io/products/pivotal-rabbitmq, lets assume we will work using pivotal-rabbitmq-server-3.5.1

2- extract the downloaded file to the installation location the tar.gz package will be self contained except for erlang liberaries, you need to install erlang from epel erlang yum / dnf repositories.

2- Once installed, we need to edit some config files to allow the rabbitmq server to run:

${RABBITMQ_HOME}/sbin/rabbitmq-defaults :
we need to set the variable ${RABBITMQ_HOME}, this will automatically populate the rest of the variables:

[root@khofo02 sbin]# cat rabbitmq-defaults |grep -v "^#"
RABBITMQ_HOME=/root/pivotal-rabbitmq-server-3.5.1/
SYS_PREFIX=${RABBITMQ_HOME}
ERL_DIR=
CLEAN_BOOT_FILE=start_clean
SASL_BOOT_FILE=start_sasl
BOOT_MODULE="rabbit"
CONFIG_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq
LOG_BASE=${SYS_PREFIX}/var/log/rabbitmq
MNESIA_BASE=${SYS_PREFIX}/var/lib/rabbitmq/mnesia
ENABLED_PLUGINS_FILE=${SYS_PREFIX}/etc/rabbitmq/enabled_plugins
PLUGINS_DIR="${RABBITMQ_HOME}/plugins"
IO_THREAD_POOL_SIZE=64
CONF_ENV_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq-env.conf
[root@khofo02 sbin]#




${RABBITMQ_HOME}/etc/rabbitmq/rabbitmq-env.conf :
We need to set the NODENAME env. variable, this is not mandatory, but it helps if we are going to run multiple RabbitMQ nodes on this host.
Also any other env. variable that needs to be defined can be placed in this file:
 
[root@khofo02 sbin]# cat ../etc/rabbitmq/rabbitmq-env.conf
#example rabbitmq-env.conf file entries
#Rename the node
NODENAME=rabbitx1
#Config file location and new filename without the config extention
#CONFIG_FILE=/root/pivotal-rabbitmq-server-3.5.1/etc/rabbitmq/rabbitmq


[root@khofo02 sbin]#


Note that the CONFIG_FILE variable is already set in the rabbitmq-defaults, so we are commenting this out in the rabbitmq-env.conf.


Note that any reference to rabbitMQ config file is done without its .config extension, thus the variable will have a value:

ONFIG_FILE=/root/pivotal-rabbitmq-server-3.5.1/etc/rabbitmq/rabbitmq


While the file on the system is:

/root/pivotal-rabbitmq-server-3.5.1/etc/rabbitmq/rabbitmq.config



${RABBITMQ_HOME}/etc/rabbitmq/rabbitmq.config :
This is the main config file for RabbitMQ, it will contain most of the configs for rabbitMQ and its plugins.
For this example we are putting configurations for management and stomp plugins.

[
    {rabbit, [{tcp_listeners, [5674]}, {loopback_users, []}]},
    {rabbitmq_management, [{listener, [{port,15674}, {ip, "0.0.0.0"}]}]},
    {rabbitmq_stomp, [{tcp_listeners, [{"0.0.0.0",61614}]},
              {default_user, [{login, "guest"},
                                      {passcode, "guest"}
                                     ]
                      },
                      {implicit_connect, true}
                     ]}
  ].

Here we are changing the default port from 5672 to 5674 using the rabbit tcp_listeners parameter.
Since the default for the guest user it only to login locally, we set the loopback_users parameter to an empty string "[]" this will allow guest to login from all interfaces.

Also to force the management plugin to listen on all interfaces we use the ip parameter set to 0.0.0.0, also the port number is changed to 15674 using listener port parameter.

For stomp we are defining a set of parameters:
  • tcp_listener on all interfaces and port is 61614, default port is 61613.
  • default_user: login as guest, password as guest.
  • implicit_connect is set to true.

Now we are ready to enable the plugins that we are going to use.
To do this we go to ${RABBITMQ_HOME}/sbin and run the commands below.
it will git output similar to the below:


[root@khofo02 sbin]# ./rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
  mochiweb
  webmachine
  rabbitmq_web_dispatch
  amqp_client
  rabbitmq_management_agent
  rabbitmq_management

Applying plugin configuration to rabbitx1@khofo02... failed.
 * Could not contact node rabbitx1@khofo02.
   Changes will take effect at broker restart.
 * Options: --online  - fail if broker cannot be contacted.
            --offline - do not try to contact broker.
[root@khofo02 sbin]# ./rabbitmq-plugins enable rabbitmq_stomp
The following plugins have been enabled:
  rabbitmq_stomp

Applying plugin configuration to rabbitx1@khofo02... failed.
 * Could not contact node rabbitx1@khofo02.
   Changes will take effect at broker restart.
 * Options: --online  - fail if broker cannot be contacted.
            --offline - do not try to contact broker.
[root@khofo02 sbin]#

Once we enable the plugins, we can start rabbitMQ server, from ${RABBITMQ_HOME}/sbin run:

[root@khofo02 sbin]# ./rabbitmq-server -detached
Warning: PID file not written; -detached was passed.
[root@khofo02 sbin]#


RabbitMQ logs at: ${RABBITMQ_HOME}/var/log/rabbitmq/rabbitx1.log
Where rabbitx1 is the name we gave above for the rabbit instance.
If all goes well, the log will looks like this:
[root@khofo02 sbin]# tail "/root/springsource/pivotal-rabbitmq-server-3.5.6//var/log/rabbitmq/rabbitx1.log"

=INFO REPORT==== 26-Jul-2016::08:28:38 ===
Server startup complete; 7 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * webmachine
 * mochiweb
 * rabbitmq_management_agent
 * rabbitmq_stomp
 * amqp_client
[root@khofo02 sbin]#

Finally to stop RabbitMQ, we run the command:

[root@khofo02 sbin]# ./rabbitmqctl stop
Stopping and halting node rabbitx1@khofo02 ...
[root@khofo02 sbin]#


I will discuss the clustering of RabbitMQ in another post.


Monday 25 July 2016

Mysql root password reset

I have been seeing this many times over my career, the need to reset mysql root password.

This could be done in 2 ways.

1- you know the root password and want to reset it:

This is a simple case, we need not bring down the DB, this can be done from any terminal that can login as root to mysql, usually the local server running mysqld.
to do this do:

login to mysql client with the password we have:

# mysql -u root -p

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';

or 

mysql> UPDATE mysql.user
    SET authentication_string = PASSWORD('MyNewPass'), password_expired = 'N'
    WHERE User = 'root' AND Host = 'localhost';
FLUSH PRIVILEGES;




2- you lost the root password and want to reset it:
This would need to use the generic way to do the reset:
To do it run:

stop mysql services

# systemctl stop mysql.service

then bring up mysql with --skip-grant-tables & --skip-networking this is to prevent external users to connect to our server while all privileges are granted to all users.

# mysqld --skip-grant-tables --skip-networking --user=mysql --pid=/var/run/mysqld/mysqld.pid

the login as root:

# mysql -u root

mysql>FLUSH PRIVILAGES;
mysql>ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';

Running "FLUSH PRIVILAGES" is to reload the whole permission system in mysql so that we can change the root password.
Once this is done, we need to kill the skip-grant-tables instance and then bring up mysql normally.

The MySQL documentation describes this quite well at http://dev.mysql.com/doc/refman/5.7/en/resetting-permissions.html


Tuesday 19 July 2016

Simple apache archiva crawler

I had a requirement to be able to identify new application deployed on the fly.
The applications are pushed to an apache archiva repository that will be used to store the application snapshots.

Also the requirement mentions that we need to identify all new released apps automatically for older deployed apps.

To be able to accomplish this, we need to have a simple crawler to go and looks for all the latest war files stored in archiva.

below is a simple script to do this:

ARCHIVA_BASE="http://archiva:8080/archiva/repository/snapshots/com/sherif/"
BASEURLS=`curl -s ${ARCHIVA_BASE} |grep "<li><a href=" |cut -d"\"" -f2 |grep -v "\.\./"`
for pUrl in `echo ${BASEURLS}`
do
        #echo ${ARCHIVA_BASE}${pUrl}
        LAST_SNAPSHOT=`curl -s ${ARCHIVA_BASE}${pUrl} | grep "<li><a href=" |cut -d"\"" -f2|egrep -v "xml|\.\./"|egrep "[0-9]+.[0-9]+.[0-9]+" |tr "." ","|sort -n -t"," -k1,2|tr "," "."| tail -1`
        if [ "x${LAST_SNAPSHOT}" = "x" ]
        then
                continue
        else
                #echo ${ARCHIVA_BASE}${pUrl}${LAST_SNAPSHOT}
                WAR=`curl -s ${ARCHIVA_BASE}${pUrl}${LAST_SNAPSHOT} | grep "<li><a href=" |cut -d"\"" -f2 |egrep ".war$"|egrep -v "md5|sha|pom"`
                if [ "x${WAR}" = "x" ]
                then
                        continue
                else
                        echo ${ARCHIVA_BASE}${pUrl}${LAST_SNAPSHOT}${WAR}
                fi


        fi
        #read
done


The script uses some assumptions as per the requirement:

1- the application folders are immediately under the archiva base URL above.
2- only war files will be deployed, using a small modification we can also get other file types.
3- the release snapshot folders are immiadiately under the application folders
4- the release snapshot folders have the format "11.22.33{Anystring}" thus conatins 3 number sections and those are used for sorting.
5- the war file are immediate found under the snapshot folder.
6- Each snapshot folder contain only 1 war file.


Wednesday 13 July 2016

Fixing Docker SSL problems in Centos

I have been blocked by SSL issues and unable to use docker for quite a while now.
I did some reading and found out some useful info in OpenSSL documentation and on a user blog about docker.

The issue is my company uses its own SSL cert to re-encrypt all SSL traffic after it is filtered in the company internal network.
The Root CA cert is not trusted by all browsers and tools thus needs to be imported to make your life less painful :)

To import a cert on Centos we need to save it under the below path:

/usr/share/pki/ca-trust-source/anchors

The anchors folder should contain certs that are in PEM format.
Once the cert is saved, you need to run the command:

 update-ca-trust

This will update the system wide trust store at:

/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

This file is linked under:

/etc/ssl/certs

Once this is done.

you need to follow the steps in this wiki:
http://richmegginson.livejournal.com/27936.html

In my case, the steps are:


 cd /etc/docker/certs.d
 mkdir dseab33srnrn.cloudfront.net
 cd dseab33srnrn.cloudfront.net
 ln -s /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
 systemctl restart docker

Each time a docker pull is needed from a certain web host, we need to execute the last part of the steps so that docker can trust the cert.

Thanks for Rich Meggisnson for solving this issue by looking up the docker code.

Monday 11 July 2016

Rundeck user management

To add a new user to rundeck, we need to edit the file:
rundeck/server/config/realm.properties

the file looks like this:

$ cat realm.properties
#
# This file defines users passwords and roles for a HashUserRealm
#
# The format is
#  <username>: <password>[,<rolename> ...]
#
# Passwords may be clear text, obfuscated or checksummed.  The class
# org.mortbay.util.Password should be used to generate obfuscated
# passwords or password checksums
#
# This sets the temporary user accounts for the Rundeck app
#
admin:admin,user,admin,api_token_group
user:user,user
sherif:sherif,otherusers,user,api_token_group


To Authorize the user to have certain privilages, we create a new policy file at:
rundeck/etc/otherusers.aclpolicy

$ cat otherusers.aclpolicy

description: Limited user access for adm restart action
context:
  project: 'someproj.*'
for:
  resource:
    - allow: [read]
  job:
    - allow: [read,run,kill]
  node:
    - allow: [read,run,refresh]
by:
  group: [otherusers]
---
description: Limited user
context:
  application: 'rundeck'
for:
  #resource:
   # - equals:
    #    kind: system
    #  allow: [read] # allow read of system info
  project:
    - match:
        name: 'someproj.*'
      allow: [read]
by:
  group: [otherusers]
$


This policy will grant the group "otherusers" limited access to just be able to see run and kill jobs for the projects matching "someproj.*" pattern.
This policy is a modified copy from admin policy.

Both the policy and the realm files will be loaded automatically by rundeck, no restart is required.

HAproxy configuration as an HTTP router / balancer

HAproxy is a very flexible reliable TCP proxy and balancer middleware.
It can be used as a generic TCP proxy / port mapper or as a TCP load balancer.
It also supports using HTTP protocol mode where it is able to work as an http proxy server and loadbalancer.

In this post i will focus on the http mode, it is used more for implementing web proxies providing high availability for web applications. below is a sample config for HAproxy as an http proxy and router:


#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#   http://haproxy.1wt.eu/download/1.3/doc/configuration.txt
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 
    maxconn     400000
    user        sherif
    group       sherif
    daemon
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode        http
    log         global
    option      dontlognull
    option      httpclose
    option      httplog
    option      forwardfor
    option      redispatch
    timeout connect 100000 # default 10 second time out if a backend is not found
    timeout client 600000
    timeout server 600000
    maxconn     600000
    retries     3
    errorfile 503 /etc/haproxy/errors/503.http
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main vardamire:8080
acl app1_url                     path_beg     /app1
acl test_url                    path_beg    /tst
acl app2_url                    path_dir    app2
###################################################################
use_backend app1_8080                        if   app1_url
use_backend app2_8081                        if   test_url app2_url   
use_backend default_services
###################################################################
backend app1_8080
    balance     roundrobin
    option httpchk GET /app1/alive
    cookie JSESSIONID prefix nocache
    server      app1    server1:8080 check cookie a1
    server      app2    server2:8080 check cookie a2
    server      app3    server3:8080 check cookie a3 


backend app2_8081
    balance     roundrobin
    option httpchk GET /tst/app2/alive
    server      app1     server2:8081 check
    server      app2     server3:8081 check
 

backend default_services
   server app1 web.toplevelbalancer:8080 check

listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth admin:admin


The above config is an example for doing http routing and balancing based on URL patterens.
Also it shows how HAproxy can handle session stickiness.

The keyword acl defines a url pattern using either pathbeg (path begins) or path_dir (path directory portion).
Then the backend keyword is used to define a couple of application backends to HAproxy, those will do the actual serving of the content.
the balance keyword is used to tell HAproxy to do a round robin load balancing between the defined servers, also it adds the option httpchk which will do an http check on the given URI for each defined server to determine if it is up or not.
Also cookie keyward is used to append a part to the JSESSIONID cookie and have it checked by HAproxy to be able to maintain session stickiness. HAproxy will prefix JSESSIONID with the cookie defined in each server and thus will be able to keep track which session goes to which server.

Lastly we are enabling HAproxy statistics so that we can monitor the status of our backends and also the stats of the requests coming to them.

This config was used successfully to route and balance 40+ services for a big project and it working fairly smooth even under load testing.

HAproxy is very light weight and can handle 10s of thousands of connections without issues.



Sunday 10 July 2016

Some SQL tips used for a puppet migration project

Puppet console stores its info in a DB, usually postgres, we needed to extract some info from that for a puppet migration project.
the tables we were looking at where nodes table and parameters table.
Those needed to be joined so we can extract all the parameters that are defined for a node as follows:


select nodes.name, parameters.key, parameters.value from nodes,parameters where
nodes.id = parameters.parameterable_id
order by nodes.name

The output of this query was exported and cleaned up with "sed" so that we can import it as a table once more.
however, puppet stores the data as key value items tied to each node name, thus if you have 3 paramters per node, you end up with 3 records with same node name and different key value pairs.

To transform this to a relational DB like table, I used this query with help of my colleague Mohamed Youssef, our Senor DB engineer:


create view nodes as (
select r.name , a.value as val1 ,r.value as val2, g.value as val3
from
(select name, value from param where key='val1') r , (select name, value from param where key='val2') a , (select name, value from param where key='val3') g
where r.name=g.name
and r.name=a.name )