RabbitMQ Clustering configurations

In this post we will see how to cluster rabbitMQ in 2 configurations.
one using same host, called a vertical cluster, and other using 2 differents hosts called the horizontal cluster.

Create Vertical clustering

1- Start RabbitMQ with differant node names and port numbers similar to below:

RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit ./rabbitmq-server -detached
RABBITMQ_NODE_PORT=5673 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15673}]" RABBITMQ_NODENAME=rabbit2 ./rabbitmq-server -detached
 
using the above env. variable form of starting up rabbitMQ vs using config file; enables us to use diffent ports and nodenames.
The config file should be empty in this case.
The above will start 2 RabbitMQ nodes on the same VM each one has its unique name and has its own uniq set of ports.
2- To enable the 2 nodes to be in a single cluster with one node as DISC and one as RAM do the below:

./rabbitmqctl -n rabbit2@beren stop_app -> stop the rabbitMQ application but node is still running on host beren
./rabbitmqctl -n rabbit2@beren join_cluster --ram rabbit@beren -> Add rabbit2 as RAM node to rabbit cluster, default is DISC
./rabbitmqctl -n rabbit2@beren start_app -> Start the rabbitMQ application
./rabbitmqctl -n rabbit@beren cluster_status -> check the cluster status should show something like this:

Cluster status of node rabbit@beren ...
[{nodes,[{disc,[rabbit@beren]},{ram,[rabbit2@beren]}]},
{running_nodes,[rabbit2@beren,rabbit@beren]},
{partitions,[]}]
...done.
 
3- to change node type from RAM to DISC do:

./rabbitmqctl -n rabbit2@beren stop_app
./rabbitmqctl -n rabbit2@beren change_cluster_node_type disc
./rabbitmqctl -n rabbit2@beren start_app
 
to change it back to RAM do:

./rabbitmqctl -n rabbit2@beren stop_app
./rabbitmqctl -n rabbit2@beren change_cluster_node_type ram
./rabbitmqctl -n rabbit2@beren start_app
Note that at least the cluster should have one disc node. Ram nodes don't offer faster response in queue performance but only offer better speed on queue creation, recommended to have at least 1 disc node on every VM to allow high availability and add ram nodes in a vertical cluster as decscribed above IF needed.

Create Horizontal clustering

To do this we need a second VM running RabbitMQ.
on the new 2nd node do the following.
1- copy the erlang cookie file from the old VM RabbitMQ user home directory to the new VM user home directory.
cookie file name is .erlang.cookie

2- go to RABBITMQ_HOME/sbin & start rabbitMQ

./rabbitmq-server -detached

3- Run the below list of commands:

./rabbitmqctl -n rabbit@vardamir stop_app
./rabbitmqctl -n rabbit@vardamir join_cluster rabbit@beren
./rabbitmqctl -n rabbit@vardamir start_app

All the queues and topics created will automatically be transfered to the new joining node.
This will create a 2 node on 2 VM cluster that will offer true HA for the rabbitMQ. queues can be configured to work in HA mode by creating a policy from the rabbitMQ admin console, the policy should containt ha-mode:all so that the queues are all working across the cluster in HA.
Also replication can be enabled in the same manner.

HAproxy config

HA proxy can be used to offer true load-balancing for a rabbitMQ cluster.
The cluster doesn't offer a common gateway by default so to benefit from it we need balancer sitting infornt of it.
In this case haproxy can help as it is very light weight and a performer.
The config for balancing 2 nodes should look like this:

# Simple configuration for an HTTP proxy listening on port 8080 on all
# interfaces and forwarding requests to a single backend "servers" with a
# single server "server1" listening on 127.0.0.1:8000

    global
    log 127.0.0.1 local0 info
    daemon
   
    defaults
    log     global
    mode     tcp
    option tcplog
    option dontlognull
    retries 3
    option redispatch
    maxconn    2000
    timeout connect 10s
    timeout client     120s
    timeout    server     120s
   
    listen rabbitmq_local_cluster 0.0.0.0:8080
   
    balance roundrobin
    server rabbitnode1    beren:5672 check inter 5000 rise 2 fall 3 #Disk Node
    server rabbitnode2 vardamir:5672 check inter 5000 rise 2 fall 3 #Disk Node
   
   
    listen private_monitoring  :8101
    mode http
    option httplog
    stats enable
    stats uri /stats
    stats refresh  30s

The above makes use of haproxy tcp mode since rabbitMQ uses amqp which is not http compatible.

Setting up RabbitMQ and its config files

This post is to document how to install rabbitMQ using the generic linux distribution that comes as a tar.gz package.

To proceed with the installation follow the below steps:
1- download latest RabbitMQ from pivotal: https://network.pivotal.io/products/pivotal-rabbitmq, lets assume we will work using pivotal-rabbitmq-server-3.5.1

2- extract the downloaded file to the installation location the tar.gz package will be self contained except for erlang liberaries, you need to install erlang from epel erlang yum / dnf repositories.

2- Once installed, we need to edit some config files to allow the rabbitmq server to run:

${RABBITMQ_HOME}/sbin/rabbitmq-defaults :
we need to set the variable ${RABBITMQ_HOME}, this will automatically populate the rest of the variables:

[root@khofo02 sbin]# cat rabbitmq-defaults |grep -v "^#"
RABBITMQ_HOME=/root/pivotal-rabbitmq-server-3.5.1/
SYS_PREFIX=${RABBITMQ_HOME}
ERL_DIR=
CLEAN_BOOT_FILE=start_clean
SASL_BOOT_FILE=start_sasl
BOOT_MODULE="rabbit"
CONFIG_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq
LOG_BASE=${SYS_PREFIX}/var/log/rabbitmq
MNESIA_BASE=${SYS_PREFIX}/var/lib/rabbitmq/mnesia
ENABLED_PLUGINS_FILE=${SYS_PREFIX}/etc/rabbitmq/enabled_plugins
PLUGINS_DIR="${RABBITMQ_HOME}/plugins"
IO_THREAD_POOL_SIZE=64
CONF_ENV_FILE=${SYS_PREFIX}/etc/rabbitmq/rabbitmq-env.conf
[root@khofo02 sbin]#



${RABBITMQ_HOME}/etc/rabbitmq/rabbitmq-env.conf :
We need to set the NODENAME env. variable, this is not mandatory, but it helps if we are going to run multiple RabbitMQ nodes on this host.
Also any other env. variable that needs to be defined can be placed in this file:
 
[root@khofo02 sbin]# cat ../etc/rabbitmq/rabbitmq-env.conf
#example rabbitmq-env.conf file entries
#Rename the node
NODENAME=rabbitx1
#Config file location and new filename without the config extention
#CONFIG_FILE=/root/pivotal-rabbitmq-server-3.5.1/etc/rabbitmq/rabbitmq

[root@khofo02 sbin]#

Note that the CONFIG_FILE variable is already set in the rabbitmq-defaults, so we are commenting this out in the rabbitmq-env.conf.


Note that any reference to rabbitMQ config file is done without its .config extension, thus the variable will have a value:

ONFIG_FILE=/root/pivotal-rabbitmq-server-3.5.1/etc/rabbitmq/rabbitmq


While the file on the system is:

/root/pivotal-rabbitmq-server-3.5.1/etc/rabbitmq/rabbitmq.config



${RABBITMQ_HOME}/etc/rabbitmq/rabbitmq.config :
This is the main config file for RabbitMQ, it will contain most of the configs for rabbitMQ and its plugins.
For this example we are putting configurations for management and stomp plugins.

[
    {rabbit, [{tcp_listeners, [5674]}, {loopback_users, []}]},
    {rabbitmq_management, [{listener, [{port,15674}, {ip, "0.0.0.0"}]}]},
    {rabbitmq_stomp, [{tcp_listeners, [{"0.0.0.0",61614}]},
              {default_user, [{login, "guest"},
                                      {passcode, "guest"}
                                     ]
                      },
                      {implicit_connect, true}
                     ]}
  ].

Here we are changing the default port from 5672 to 5674 using the rabbit tcp_listeners parameter.
Since the default for the guest user it only to login locally, we set the loopback_users parameter to an empty string "[]" this will allow guest to login from all interfaces.

Also to force the management plugin to listen on all interfaces we use the ip parameter set to 0.0.0.0, also the port number is changed to 15674 using listener port parameter.

For stomp we are defining a set of parameters:

• tcp_listener on all interfaces and port is 61614, default port is 61613.
• default_user: login as guest, password as guest.
• implicit_connect is set to true.

Now we are ready to enable the plugins that we are going to use.
To do this we go to ${RABBITMQ_HOME}/sbin and run the commands below.
it will git output similar to the below:


[root@khofo02 sbin]# ./rabbitmq-plugins enable rabbitmq_management
The following plugins have been enabled:
  mochiweb
  webmachine
  rabbitmq_web_dispatch
  amqp_client
  rabbitmq_management_agent
  rabbitmq_management

Applying plugin configuration to rabbitx1@khofo02... failed.
 * Could not contact node rabbitx1@khofo02.
   Changes will take effect at broker restart.
 * Options: --online  - fail if broker cannot be contacted.
            --offline - do not try to contact broker.
[root@khofo02 sbin]# ./rabbitmq-plugins enable rabbitmq_stomp
The following plugins have been enabled:
  rabbitmq_stomp

Applying plugin configuration to rabbitx1@khofo02... failed.
 * Could not contact node rabbitx1@khofo02.
   Changes will take effect at broker restart.
 * Options: --online  - fail if broker cannot be contacted.
            --offline - do not try to contact broker.
[root@khofo02 sbin]#

Once we enable the plugins, we can start rabbitMQ server, from ${RABBITMQ_HOME}/sbin run:

[root@khofo02 sbin]# ./rabbitmq-server -detached
Warning: PID file not written; -detached was passed.
[root@khofo02 sbin]#

RabbitMQ logs at: ${RABBITMQ_HOME}/var/log/rabbitmq/rabbitx1.log
Where rabbitx1 is the name we gave above for the rabbit instance.
If all goes well, the log will looks like this:
[root@khofo02 sbin]# tail "/root/springsource/pivotal-rabbitmq-server-3.5.6//var/log/rabbitmq/rabbitx1.log"

=INFO REPORT==== 26-Jul-2016::08:28:38 ===
Server startup complete; 7 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * webmachine
 * mochiweb
 * rabbitmq_management_agent
 * rabbitmq_stomp
 * amqp_client
[root@khofo02 sbin]#

Finally to stop RabbitMQ, we run the command:

[root@khofo02 sbin]# ./rabbitmqctl stop
Stopping and halting node rabbitx1@khofo02 ...
[root@khofo02 sbin]#

I will discuss the clustering of RabbitMQ in another post.