Showing posts with label docker. Show all posts
Showing posts with label docker. Show all posts

Sunday, 13 September 2020

Using Docker registry

One of the esiest ways to deploy applications is to ship your application as a docker container.
Application build process can include a step for creating and building a docker image for the target app.
Docker images then needs to be pushed to a docker registry, so that the deployment step can use that image to deploy the application.
In order for the pipeline to work, one needs a docker registry to host the images, either local or public.
Docker hub offers a cloud hosted registry, which is very flexible and always available, but requires your systems to have internet access.
I tested using Docker hub registry and building a simple local registry.

Running a local docker registry is simple, its just another docker container.
sherif@Luthien:~$ cat docker_registry.sh 
 docker run -d \
  -p 5100:5000 \
  --restart=always \
  --name luthien_registry \
  -v /registry:/var/lib/registry \
  registry:2
In the above script, I am mapping the default registry port 5000 on the container to port 5100 on the host machine.
To verify if the registry is running, we use the docker ps or docker container ps command to check:
sherif@Luthien:~$ docker container ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED              STATUS              PORTS                         NAMES
26009ca9e4f4        registry:2                   "/entrypoint.sh /etc…"   About a minute ago   Up About a minute   0.0.0.0:5100->5000/tcp        luthien_registry
sherif@Luthien:~$ 
Next we need to build a test image to test the new registry, to do this we use a docker file:
sherif@Luthien:~$ cat Dockerfile 
FROM busybox
CMD echo "Hello world! This is my first Docker image."
sherif@Luthien:~$
Then we use docker build to create the image locally:
sherif@Luthien:~$ docker build -f ./Dockerfile -t busybox_hello:1.0 ./ 
Sending build context to Docker daemon  3.432GB
Step 1/2 : FROM busybox
 ---> 6858809bf669
Step 2/2 : CMD echo "Hello world! This is my first Docker image."
 ---> Running in 3a7a187ab5ba
Removing intermediate container 3a7a187ab5ba
 ---> 63213a968c8e
Successfully built 63213a968c8e
Successfully tagged busybox_hello:1.0

sherif@Luthien:~$ docker image ls -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
busybox_hello       1.0                 7415dea3e476        22 seconds ago      1.23MB
busybox             latest              6858809bf669        3 days ago          1.23MB
sherif@Luthien:~$
Then we tag the local image using the name:port/tag:version of the image to be pushed to the registry.
sherif@Luthien:~$ docker tag busybox_hello:1.0 luthien:5100/busybox_hello:1.0 

sherif@Luthien:~$ docker image ls -a
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
busybox_hello                1.0                 7415dea3e476        23 minutes ago      1.23MB
luthien:5100/busybox_hello   1.0                 7415dea3e476        23 minutes ago      1.23MB
busybox                      latest              6858809bf669        3 days ago          1.23MB
sherif@Luthien:~$
Then we push the locally tagged image to the registry using docker push:
sherif@Luthien:~$ docker push luthien:5100/busybox_hello:1.0
The push refers to repository [luthien:5100/busybox_hello]
be8b8b42328a: Pushed 
1.0: digest: sha256:d7c348330e06aa13c1dc766590aebe0d75e95291993dd26710b6bbdd671b30d1 size: 527
sherif@Luthien:~$
In order to confirm if the image was pushed, we use the docker registry rest API to query for our new image:
sherif@Luthien:~$ curl -LX GET http://luthien:5100/v2/_catalog
{"repositories":["busybox_hello"]}
sherif@Luthien:~$ curl -LX GET http://luthien:5100/v2/busybox_hello/tags/list
{"name":"busybox_hello","tags":["1.0"]}
sherif@Luthien:~$

If we want to push to an exsiting docker hub repository, we follow a simpler process.
We just need to build the image with the repository tag, do a docker login then do a docker push.
This should push the image as a version tag into the existing repository.
sherif@Luthien:~$ docker build -t sfattah/sfattah_r:1.0 ./

sherif@Luthien:~$ docker login 
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: sfattah
Password: 
Login Succeeded
sherif@Luthien:~$ docker push  sfattah/sfattah_r:1.0
The push refers to repository [docker.io/sfattah/sfattah_r]
be8b8b42328a: Mounted from library/busybox 
1.0: digest: sha256:d7c348330e06aa13c1dc766590aebe0d75e95291993dd26710b6bbdd671b30d1 size: 527
sherif@Luthien:~$

Tuesday, 5 May 2020

Installing Keycloak using Docker with Nginx as Reverse proxy

I wanted to test setting up an on premise identity provider and see if I can build my own SSO solution.
Found that the easiest way to do this is to use Keycloak.
Keycloak is a feature rich identity access management system that supports various integration protocols, among those, SAML 2.0 and OpenID connect.

Keycloak can also connect to existing IDPs and LDAP user directories to allow more flexible setups.

To install Keycloak using you can download it from its official download page: https://www.keycloak.org/downloads or use a docker image.
To use docker, we need to pull the official image from the official docker repo:

docker pull jboss/keycloak

Once we have the image, you can run keycloak using below simple docker run:

docker run -p 8080:8080 -p 8443:8443 jboss/keycloak
docker exec <CONTAINER> /opt/jboss/keycloak/bin/add-user-keycloak.sh -u <USERNAME> -p <PASSWORD>

This will start keycloak and expose port 8080 and 8443 to the docker host, then using docker exec, we run the script add-user-keycloak.sh to add an admin user.

In our use case here, we will start keycloak using a bigger set of options to allow it to  interact with a docker network, import a realm, that was saved from a previous setup, add admin user directly from docker run command and add reverse proxy forwarding options.

But first, we create a docker bridge network:

[root@Fingon ~]# docker network create  --subnet=172.28.0.0/16 --gateway=172.28.0.1 dcnet


And then the docker run script looks like this:

docker run \
-e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin \
-e PROXY_ADDRESS_FORWARDING=true \
-e KEYCLOAK_IMPORT=/tmp/realm-export.json \
-v /tmp/realm-export.json:/tmp/realm-export.json \
--network dcnet \
--hostname keycloak2 \
--ip 172.28.0.13 \
--add-host fingon:172.17.0.1 \
--add-host fingon:172.28.0.1 \
--add-host dcnode2:172.28.0.12 \
--add-host dcnode1:172.28.0.11 \
--add-host keycloak2:172.28.0.13 \
--dns 8.8.4.4 \
--dns 8.8.8.8 \
--dns 172.28.0.1 \
--name="keycloak2" -d -p 8181:8080 -p 9443:8443 jboss/keycloak



And once the container us up, we can see it with docker ps:

[root@Fingon ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES                                                                                                                
3402c1b4163e        jboss/keycloak      "/opt/jboss/tools/do…"   21 hours ago        Up 3 seconds        0.0.0.0:8181->8080/tcp, 0.0.0.0:9443->8443/tcp   keycloak2                                                                                                            
[root@Fingon ~]#



We can now access keycloak from host machine using the URL http://localhost:8181/auth/ or https://localhost:9443/auth/

To expose keycloak to outside network and use proper SSL configuration, we then need to use a reverse proxy, in this case we use Nginx.

The Nigix configuration will look like this:

[root@Fingon nginx]# cat nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$http_authorization"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        return 301 https://fingon$request_uri;
    }

# Settings for a TLS enabled server.
#
    server {
        listen       443 ssl http2 default_server;
        listen       [::]:443 ssl http2 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        ssl_certificate "/root/cert.crt";
        ssl_certificate_key "/root/key.key";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }
               
        location /auth/ {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_pass http://localhost:8181/auth/;

        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

}

[root@Fingon nginx]#


Now the UI looks like below:




You can access keycloak on https only using the URL https://fingon/auth.
Any requests to http will be redirected to https.
On Nginx, we need to set the host, x-forwarded-for and x-forwarded-proto headers so that keycloak identifies it is working behind a reverse proxy and does a proper redirection.
More information can be found in keycloak documentation.

Now we are ready to do more keycloak configurations ☺











Wednesday, 13 July 2016

Fixing Docker SSL problems in Centos

I have been blocked by SSL issues and unable to use docker for quite a while now.
I did some reading and found out some useful info in OpenSSL documentation and on a user blog about docker.

The issue is my company uses its own SSL cert to re-encrypt all SSL traffic after it is filtered in the company internal network.
The Root CA cert is not trusted by all browsers and tools thus needs to be imported to make your life less painful :)

To import a cert on Centos we need to save it under the below path:

/usr/share/pki/ca-trust-source/anchors

The anchors folder should contain certs that are in PEM format.
Once the cert is saved, you need to run the command:

 update-ca-trust

This will update the system wide trust store at:

/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

This file is linked under:

/etc/ssl/certs

Once this is done.

you need to follow the steps in this wiki:
http://richmegginson.livejournal.com/27936.html

In my case, the steps are:


 cd /etc/docker/certs.d
 mkdir dseab33srnrn.cloudfront.net
 cd dseab33srnrn.cloudfront.net
 ln -s /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
 systemctl restart docker

Each time a docker pull is needed from a certain web host, we need to execute the last part of the steps so that docker can trust the cert.

Thanks for Rich Meggisnson for solving this issue by looking up the docker code.

Monday, 30 May 2016

Docker command tips

This post is to allow me remember how to use docker :)
since i am getting older i tend to forget new syntax more than before !!

0- Docker pull:
Docker pull will fetch a docker image from the default docker registry.
eg:
docker pull redis:latest
 
You can browse the official docker repository at: https://hub.docker.com/explore/
Another way to look for docker images is to use docker search as below:
[root@fingolfin stock_apache_docker]# docker search mysql
INDEX       NAME                                 DESCRIPTION                                     STARS     OFFICIAL   AUTOMATED
docker.io   docker.io/mysql                      MySQL is a widely used, open-source relati...   2409      [OK]     
docker.io   docker.io/mysql/mysql-server         Optimized MySQL Server Docker images. Crea...   149                  [OK]
docker.io   docker.io/centurylink/mysql          Image containing mysql. Optimized to be li...   45                   [OK]
docker.io   docker.io/sameersbn/mysql                                                            35                   [OK]
docker.io   docker.io/google/mysql               MySQL server for Google Compute Engine          16                   [OK]
docker.io   docker.io/appcontainers/mysql        Centos/Debian/Ubuntu Based Customizable My...   7                    [OK]
docker.io   docker.io/marvambass/mysql           MySQL Server based on Ubuntu 14.04              6                    [OK]
docker.io   docker.io/alterway/mysql             Docker Mysql                                    2                    [OK]
docker.io   docker.io/azukiapp/mysql             Docker image to run MySQL by Azuki - http:...   2                    [OK]
docker.io   docker.io/drupaldocker/mysql         MySQL for Drupal                                2                    [OK]
docker.io   docker.io/yfix/mysql                 Yfix docker built mysql                         2                    [OK]
docker.io   docker.io/bahmni/mysql               Mysql container for bahmni.  Contains the ...   1                    [OK]
docker.io   docker.io/frodenas/mysql             A Docker Image for MySQL                        1                    [OK]
docker.io   docker.io/ivories/mysql              mysql                                           1                    [OK]
docker.io   docker.io/phpmentors/mysql           MySQL server image                              1                    [OK]
docker.io   docker.io/sin30/mysql                MySQL images with my own config files.          1                    [OK]
docker.io   docker.io/akilli/mysql               akilli/base based MySQL image                   0                    [OK]
docker.io   docker.io/cloudposse/mysql           Improved `mysql` service with support for ...   0                    [OK]
docker.io   docker.io/dockerizedrupal/mysql      docker-mysql                                    0                    [OK]
docker.io   docker.io/lancehudson/docker-mysql   MySQL is a widely used, open-source relati...   0                    [OK]
docker.io   docker.io/livingobjects/mysql        MySQL                                           0                    [OK]
docker.io   docker.io/nanobox/mysql              MySQL service for nanobox.io                    0                    [OK]
docker.io   docker.io/projectomakase/mysql       Docker image for MySQL                          0                    [OK]
docker.io   docker.io/tozd/mysql                 MySQL (MariaDB fork) Docker image.              0                    [OK]
docker.io   docker.io/vukor/mysql                Build for MySQL. Project available on http...   0                    [OK]
[root@fingolfin stock_apache_docker]#
Also there is a useful link to how to setup a local registry: https://docs.docker.com/registry/deploying/
 
1- Docker rm:
docker rm is used to remove the docker containers, for this to work we need to list all the containers that are created in our docker system:
to do this use:

docker ps -a |cut -d" " -f1|tail -n +2

then to remove all the containers run this one line script:

docker rm `docker ps -a |cut -d" " -f1|tail -n +2`

This will remove all the containers :)

2- Docker rmi:
This command is used to remove docker images, i am using this to get red of the temp and unused ones to save disk space:

[root@fingolfin stock_apache_docker]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
                            508a8098b89d        17 minutes ago      282.3 MB
                            68cb22da5789        41 minutes ago      282.3 MB
                            fa8600453d4f        8 hours ago         282.3 MB
                            1324b4fad9fc        8 hours ago         282.3 MB
                            eb9d45dd81cd        8 hours ago         282.3 MB
                            cbf901a1a88e        8 hours ago         282.3 MB
                            b16a988c6668        8 hours ago         282.3 MB
                            e2ac7f0940f9        8 hours ago         282.3 MB
                            e74dccbf3e7d        24 hours ago        310.8 MB
                            b8cae09ad3bf        25 hours ago        196.7 MB
docker.io/ubuntu    latest              8444cb1ed763        2 days ago          122 MB
docker.io/centos    centos6             d487f1b804de        12 days ago         194.5 MB
docker.io/centos    latest              8c59c0a396b7        12 days ago         196.7 MB
[root@fingolfin stock_apache_docker]#


then:

[root@fingolfin stock_apache_docker]# docker rmi 68cb22da5789 fa8600453d4f 1324b4fad9fc eb9d45dd81cd cbf901a1a88e b16a988c6668 e2ac7f0940f9 e74dccbf3e7d b8cae09ad3bf
Deleted: 68cb22da57893d493c9a0983cc4892b33241766f2fcbbdfd752d191352bdd751
Deleted: c0fc7de087b97f1c75bc189785b9861999fafa7aef05aeb5ed90b23a5bd387d2
Deleted: fa8600453d4f95758d739cf017cec8f6ee4f50b27d4c3430f1859f414aed04fc
 . . . 
 . . . 


3- Docker run:
This is a sample command to run an apache container and map container port 80 to the host port 9090.
Also we use volumes feature of docker to mount host volumes on the container for the apache htdocs /var/www/http and logs at /var/log/httpd:

command:

docker run -p 9090:80 -v /root/docker_stage/stock_apache_docker/html:/var/www/html  -v /root/docker_stage/stock_apache_docker/logs:/var/log/httpd 68cb22da5789

you can put any content on the host folder and docker apache will pick it up. also you can collect the apache logs from the host folder with a simple shell script !

Another form of docker run is to create a shell and actually execute commands on the docker container affecting the image directly, this needs to allocate a terminal -t and be interactive -i as below:

[root@fingolfin logs]# docker run -t -i centos:centos6 /bin/bash
Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
[root@204005540b83 /]# hostname
204005540b83
[root@204005540b83 /]# cat /etc/redhat-release
CentOS release 6.7 (Final)
[root@204005540b83 /]# exit
[root@fingolfin logs]#

4- Docker build:
Docker build will process a prefined docker file and execute the steps in it one after the other using containers resulting from set n-1 to do step n.
This makes it easy to create custom containers as needed.
below is a simple dockerfile for apache:

[root@fingolfin stock_apache_docker]# cat dockerfile
FROM centos:centos6

RUN yum -y --nogpgcheck install httpd

VOLUME /var/www/html
VOLUME /var/log/httpd


EXPOSE 80

ENTRYPOINT /usr/sbin/httpd -D FOREGROUND
[root@fingolfin stock_apache_docker]#

 
Note that in the dockerfile we need to suppress all user interaction and also make sure that the entrypoint and or CMD commands are running in foreground, if they run as a service something else would need to block the container from exiting, say using a tail -f on a none rotating log . .

Also note the difference between RUN and ENTRYPOINT/CMD, run will execute its command in build time and its action will take effect at build time, in this case its a yum install, it could also be an mkdir or running any setup script that affects the image.

ENTRYPOINT/CMD will not run at build time, instead, it will run when a container is initialized from  the image, say when we use a docker run command. ENTRYPOINT should only appear once and thus most of the time holds the purpose of the image, in our case running apache.
CMD can appear multiple times to run commands and provide arguments to the entry point command.

It’s always better to use 1 container per piece of software, makes things much easier.