How to test your system infrastructure - Part 1

In this post I need to document various ways to test parts of an application infrastructure, mainly how to check CPU usage, how fast is disk IO, how fast is network infrastructure between 2 nodes, all tests assume Linux based infrastructure.

CPU:

On Linux the easiest way to check how much CPU is being used is using the top command:

top is an interactive command, clicking 1 while top is running, it will print the CPU usage per core.
top can also be run in none interactive mode as needed:

sherif@fingolfin:~$ top -b -n 1 |head
top - 14:22:23 up 52 min,  1 user,  load average: 0,29, 0,12, 0,06
Tasks: 175 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,5 us,  0,3 sy,  0,0 ni, 98,9 id,  0,2 wa,  0,0 hi,  0,1 si,  0,0 st
KiB Mem :  6072348 total,  4880976 free,   434896 used,   756476 buff/cache
KiB Swap:  2097148 total,  2097148 free,        0 used.  5398192 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2063 root      20   0  420504  94464  33488 S   2,3  1,6   0:25.45 Xorg
    1 root      20   0  225232   9024   6748 S   0,0  0,1   0:02.70 systemd
    2 root      20   0       0      0      0 S   0,0  0,0   0:00.00 kthreadd
sherif@fingolfin:~$

Another way to report on CPU usage is using iostat command:

[root@feanor ~]# iostat -c
Linux 3.10.0-862.el7.x86_64 (feanor)    05/16/2020      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.23    0.00    0.28    0.07    0.00   99.42

[root@feanor ~]#

One other way to benchmark the CPU execution on the system is to use the sysbench package as below:

sherif@fingolfin:~$ time sysbench --test=cpu --threads=6 run
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
sysbench 1.0.11 (using system LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 6
Initializing random number generator from current time
Prime numbers limit: 10000
Initializing worker threads...
Threads started!
CPU speed:
    events per second:  3886.37
General statistics:
    total time:                          10.0011s
    total number of events:              38872
Latency (ms):
         min:                                  0.62
         avg:                                  1.54
         max:                                 29.13
         95th percentile:                      8.74
         sum:                              59820.54
Threads fairness:
    events (avg/stddev):           6478.6667/112.54
    execution time (avg/stddev):   9.9701/0.03
real    0m10,014s
user    0m29,940s
sys    0m0,012s
sherif@fingolfin:~$

The above test shows how much latency could be expected running multiple threads on the system.
More info about sysbench tool can be found in this page: https://linuxconfig.org/how-to-benchmark-your-linux-system

Memory:

To measure how fast our system memory works, we can use the small tool mbw from: https://github.com/raas/mbw.
The tools mesaures the memory bandwidth from user space, similar to what could be noticed by standard applications.
To compile the code on Centos we follow the below:

[root@feanor ~]# git clone https://github.com/raas/mbw
Cloning into 'mbw'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 89 (delta 0), reused 1 (delta 0), pack-reused 85
Unpacking objects: 100% (89/89), done.
[root@feanor ~]# cd mbw
[root@feanor mbw]# ls -ltr
total 28
-rw-r--r--. 1 root root  423 May 16 16:12 README
-rw-r--r--. 1 root root  232 May 16 16:12 Makefile
-rw-r--r--. 1 root root 1255 May 16 16:12 mbw.1
-rw-r--r--. 1 root root 1640 May 16 16:12 mbw.spec
-rw-r--r--. 1 root root 8538 May 16 16:12 mbw.c
[root@feanor mbw]# make
cc     mbw.c   -o mbw
[root@feanor mbw]# ./mbw 512
Long uses 8 bytes. Allocating 2*67108864 elements = 1073741824 bytes of memory.
Using 262144 bytes as blocks for memcpy block copy test.
Getting down to business... Doing 10 runs per test.
0       Method: MEMCPY  Elapsed: 0.08889        MiB: 512.00000  Copy: 5760.122 MiB/s
1       Method: MEMCPY  Elapsed: 0.09538        MiB: 512.00000  Copy: 5368.283 MiB/s
2       Method: MEMCPY  Elapsed: 0.09289        MiB: 512.00000  Copy: 5512.133 MiB/s
3       Method: MEMCPY  Elapsed: 0.09756        MiB: 512.00000  Copy: 5247.891 MiB/s
4       Method: MEMCPY  Elapsed: 0.09414        MiB: 512.00000  Copy: 5438.593 MiB/s
5       Method: MEMCPY  Elapsed: 0.08911        MiB: 512.00000  Copy: 5745.450 MiB/s
6       Method: MEMCPY  Elapsed: 0.08720        MiB: 512.00000  Copy: 5871.627 MiB/s
7       Method: MEMCPY  Elapsed: 0.09688        MiB: 512.00000  Copy: 5284.616 MiB/s
8       Method: MEMCPY  Elapsed: 0.09409        MiB: 512.00000  Copy: 5441.598 MiB/s
9       Method: MEMCPY  Elapsed: 0.09243        MiB: 512.00000  Copy: 5539.087 MiB/s
AVG     Method: MEMCPY  Elapsed: 0.09286        MiB: 512.00000  Copy: 5513.825 MiB/s
0       Method: DUMB    Elapsed: 0.25512        MiB: 512.00000  Copy: 2006.875 MiB/s
1       Method: DUMB    Elapsed: 0.23047        MiB: 512.00000  Copy: 2221.528 MiB/s
2       Method: DUMB    Elapsed: 0.22259        MiB: 512.00000  Copy: 2300.245 MiB/s
3       Method: DUMB    Elapsed: 0.23621        MiB: 512.00000  Copy: 2167.544 MiB/s
4       Method: DUMB    Elapsed: 0.21707        MiB: 512.00000  Copy: 2358.697 MiB/s
5       Method: DUMB    Elapsed: 0.22799        MiB: 512.00000  Copy: 2245.742 MiB/s
6       Method: DUMB    Elapsed: 0.22476        MiB: 512.00000  Copy: 2277.965 MiB/s
7       Method: DUMB    Elapsed: 0.22205        MiB: 512.00000  Copy: 2305.777 MiB/s
8       Method: DUMB    Elapsed: 0.22730        MiB: 512.00000  Copy: 2252.490 MiB/s
9       Method: DUMB    Elapsed: 0.22879        MiB: 512.00000  Copy: 2237.899 MiB/s
AVG     Method: DUMB    Elapsed: 0.22924        MiB: 512.00000  Copy: 2233.515 MiB/s
0       Method: MCBLOCK Elapsed: 0.09570        MiB: 512.00000  Copy: 5350.052 MiB/s
1       Method: MCBLOCK Elapsed: 0.10106        MiB: 512.00000  Copy: 5066.197 MiB/s
2       Method: MCBLOCK Elapsed: 0.09312        MiB: 512.00000  Copy: 5498.459 MiB/s
3       Method: MCBLOCK Elapsed: 0.09769        MiB: 512.00000  Copy: 5240.961 MiB/s
4       Method: MCBLOCK Elapsed: 0.09894        MiB: 512.00000  Copy: 5174.958 MiB/s
5       Method: MCBLOCK Elapsed: 0.09634        MiB: 512.00000  Copy: 5314.456 MiB/s
6       Method: MCBLOCK Elapsed: 0.09780        MiB: 512.00000  Copy: 5235.388 MiB/s
7       Method: MCBLOCK Elapsed: 0.09487        MiB: 512.00000  Copy: 5397.086 MiB/s
8       Method: MCBLOCK Elapsed: 0.09828        MiB: 512.00000  Copy: 5209.446 MiB/s
9       Method: MCBLOCK Elapsed: 0.09942        MiB: 512.00000  Copy: 5149.973 MiB/s
AVG     Method: MCBLOCK Elapsed: 0.09732        MiB: 512.00000  Copy: 5260.924 MiB/s
[root@feanor mbw]#

The tool is available as a Debian package.
One cool test is to see when the tool tries to allocate 4GB on the above system, that machine has only 4GB of memory, and allocating that size would drive the mbw tool to get swapped out, we can see that with multiple ways, first, the bandwidth is orders of mangitude lower:

[root@feanor mbw]# ./mbw 2048
Long uses 8 bytes. Allocating 2*268435456 elements = 4294967296 bytes of memory.
Using 262144 bytes as blocks for memcpy block copy test.
Getting down to business... Doing 10 runs per test.
0       Method: MEMCPY  Elapsed: 26.14064       MiB: 2048.00000 Copy: 78.345 MiB/s
1       Method: MEMCPY  Elapsed: 42.49331       MiB: 2048.00000 Copy: 48.196 MiB/s
2       Method: MEMCPY  Elapsed: 18.70199       MiB: 2048.00000 Copy: 109.507 MiB/s
3       Method: MEMCPY  Elapsed: 55.37665       MiB: 2048.00000 Copy: 36.983 MiB/s
4       Method: MEMCPY  Elapsed: 35.01051       MiB: 2048.00000 Copy: 58.497 MiB/s
5       Method: MEMCPY  Elapsed: 20.52362       MiB: 2048.00000 Copy: 99.787 MiB/s
6       Method: MEMCPY  Elapsed: 21.93620       MiB: 2048.00000 Copy: 93.362 MiB/s
7       Method: MEMCPY  Elapsed: 37.51056       MiB: 2048.00000 Copy: 54.598 MiB/s
8       Method: MEMCPY  Elapsed: 28.07473       MiB: 2048.00000 Copy: 72.948 MiB/s
9       Method: MEMCPY  Elapsed: 14.76706       MiB: 2048.00000 Copy: 138.687 MiB/s
AVG     Method: MEMCPY  Elapsed: 30.05353       MiB: 2048.00000 Copy: 68.145 MiB/s
0       Method: DUMB    Elapsed: 11.23370       MiB: 2048.00000 Copy: 182.309 MiB/s
1       Method: DUMB    Elapsed: 10.76112       MiB: 2048.00000 Copy: 190.315 MiB/s
2       Method: DUMB    Elapsed: 15.99955       MiB: 2048.00000 Copy: 128.004 MiB/s
3       Method: DUMB    Elapsed: 23.18597       MiB: 2048.00000 Copy: 88.329 MiB/s
4       Method: DUMB    Elapsed: 28.14035       MiB: 2048.00000 Copy: 72.778 MiB/s
5       Method: DUMB    Elapsed: 31.18035       MiB: 2048.00000 Copy: 65.682 MiB/s
6       Method: DUMB    Elapsed: 31.02135       MiB: 2048.00000 Copy: 66.019 MiB/s
7       Method: DUMB    Elapsed: 36.10925       MiB: 2048.00000 Copy: 56.717 MiB/s
8       Method: DUMB    Elapsed: 51.37134       MiB: 2048.00000 Copy: 39.867 MiB/s
9       Method: DUMB    Elapsed: 60.84004       MiB: 2048.00000 Copy: 33.662 MiB/s
AVG     Method: DUMB    Elapsed: 29.98430       MiB: 2048.00000 Copy: 68.302 MiB/s
0       Method: MCBLOCK Elapsed: 67.50246       MiB: 2048.00000 Copy: 30.340 MiB/s
1       Method: MCBLOCK Elapsed: 74.09162       MiB: 2048.00000 Copy: 27.641 MiB/s
2       Method: MCBLOCK Elapsed: 77.48624       MiB: 2048.00000 Copy: 26.430 MiB/s
3       Method: MCBLOCK Elapsed: 75.32009       MiB: 2048.00000 Copy: 27.191 MiB/s
4       Method: MCBLOCK Elapsed: 94.43207       MiB: 2048.00000 Copy: 21.688 MiB/s
5       Method: MCBLOCK Elapsed: 96.87246       MiB: 2048.00000 Copy: 21.141 MiB/s
6       Method: MCBLOCK Elapsed: 102.09089      MiB: 2048.00000 Copy: 20.061 MiB/s
7       Method: MCBLOCK Elapsed: 95.71384       MiB: 2048.00000 Copy: 21.397 MiB/s
8       Method: MCBLOCK Elapsed: 89.24437       MiB: 2048.00000 Copy: 22.948 MiB/s
9       Method: MCBLOCK Elapsed: 103.73286      MiB: 2048.00000 Copy: 19.743 MiB/s
AVG     Method: MCBLOCK Elapsed: 87.64869       MiB: 2048.00000 Copy: 23.366 MiB/s
[root@feanor mbw]#

Using the iotop tool we can see that mbw tool is swapped out:

 And using the smem tool we can see that mbw tool is swapping:

[root@feanor mbw]# smem |head -1; smem|grep mbw
  PID User     Command                         Swap      USS      PSS      RSS
 4144 root     grep --color=auto mbw              0      140      326      704
 4005 root     ./mbw 2048                    584004  3610400  3610400  3610408
[root@feanor mbw]#


To collect system wide memory information, we can use top command or we can also use vmstat:

[root@feanor mbw]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  2 2046444 112560      0 125508 8596 5262 13055  5449 2268  827  1  9 76 14  0

[root@feanor mbw]# vmstat -s
      4043552 K total memory
      3824396 K used memory
      2936656 K active memory
       782444 K inactive memory
       107972 K free memory
            0 K buffer memory
       111184 K swap cache
      4063228 K total swap
      2026724 K used swap
      2036504 K free swap
        16716 non-nice user cpu ticks
           20 nice user cpu ticks
        89870 system cpu ticks
       926212 idle cpu ticks
       171799 IO-wait cpu ticks
            0 IRQ cpu ticks
        21806 softirq cpu ticks
            0 stolen cpu ticks
    160271685 pages paged in
     66888866 pages paged out
     26373088 pages swapped in
     16147227 pages swapped out
     27846320 interrupts
     10151325 CPU context switches
   1589638134 boot time
         4248 forks
[root@feanor mbw]#

The vmstat tools uses the kernel file /proc/meminfo which contains more information about system memory usage as can be seen below:

[root@feanor mbw]# cat /proc/meminfo
MemTotal:        4043552 kB
MemFree:         3622752 kB
MemAvailable:    3570904 kB
Buffers:               0 kB
Cached:           110172 kB
SwapCached:        52480 kB
Active:            66004 kB
Inactive:         146504 kB
Active(anon):      49768 kB
Inactive(anon):    61024 kB
Active(file):      16236 kB
Inactive(file):    85480 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4063228 kB
SwapFree:        3716148 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:         64504 kB
Mapped:            24988 kB
Shmem:              8356 kB
Slab:              79168 kB
SReclaimable:      32396 kB
SUnreclaim:        46772 kB
KernelStack:        6752 kB
PageTables:        32416 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6085004 kB
Committed_AS:    2550352 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      105404 kB
VmallocChunk:   34359537660 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      131008 kB
DirectMap2M:     4063232 kB
[root@feanor mbw]#

Another small nice tool to report memory usage on a linux system is the free tool:

[root@feanor mbw]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        235M        3.4G        8.7M        230M        3.4G
Swap:          3.9G        335M        3.5G
[root@feanor mbw]#

Here the output is similar to what we get from top, we can see how much swap is used, how much memory is used by Linux to for buffers and disk caching and how much is memory available to the system.

Installing Keycloak using Docker with Nginx as Reverse proxy

I wanted to test setting up an on premise identity provider and see if I can build my own SSO solution.
Found that the easiest way to do this is to use Keycloak.
Keycloak is a feature rich identity access management system that supports various integration protocols, among those, SAML 2.0 and OpenID connect.

Keycloak can also connect to existing IDPs and LDAP user directories to allow more flexible setups.

To install Keycloak using you can download it from its official download page: https://www.keycloak.org/downloads or use a docker image.
To use docker, we need to pull the official image from the official docker repo:

docker pull jboss/keycloak

Once we have the image, you can run keycloak using below simple docker run:

docker run -p 8080:8080 -p 8443:8443 jboss/keycloak

docker exec <CONTAINER> /opt/jboss/keycloak/bin/add-user-keycloak.sh -u <USERNAME> -p <PASSWORD>

This will start keycloak and expose port 8080 and 8443 to the docker host, then using docker exec, we run the script add-user-keycloak.sh to add an admin user.

In our use case here, we will start keycloak using a bigger set of options to allow it to  interact with a docker network, import a realm, that was saved from a previous setup, add admin user directly from docker run command and add reverse proxy forwarding options.

But first, we create a docker bridge network:

[root@Fingon ~]# docker network create  --subnet=172.28.0.0/16 --gateway=172.28.0.1 dcnet


And then the docker run script looks like this:

docker run \
-e KEYCLOAK_USER=admin -e KEYCLOAK_PASSWORD=admin \
-e PROXY_ADDRESS_FORWARDING=true \
-e KEYCLOAK_IMPORT=/tmp/realm-export.json \
-v /tmp/realm-export.json:/tmp/realm-export.json \
--network dcnet \
--hostname keycloak2 \
--ip 172.28.0.13 \
--add-host fingon:172.17.0.1 \
--add-host fingon:172.28.0.1 \
--add-host dcnode2:172.28.0.12 \
--add-host dcnode1:172.28.0.11 \
--add-host keycloak2:172.28.0.13 \
--dns 8.8.4.4 \
--dns 8.8.8.8 \
--dns 172.28.0.1 \
--name="keycloak2" -d -p 8181:8080 -p 9443:8443 jboss/keycloak


And once the container us up, we can see it with docker ps:

[root@Fingon ~]# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES                                                                                                                
3402c1b4163e        jboss/keycloak      "/opt/jboss/tools/do…"   21 hours ago        Up 3 seconds        0.0.0.0:8181->8080/tcp, 0.0.0.0:9443->8443/tcp   keycloak2                                                                                                            
[root@Fingon ~]#


We can now access keycloak from host machine using the URL http://localhost:8181/auth/ or https://localhost:9443/auth/

To expose keycloak to outside network and use proper SSL configuration, we then need to use a reverse proxy, in this case we use Nginx.

The Nigix configuration will look like this:

[root@Fingon nginx]# cat nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$http_authorization"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        return 301 https://fingon$request_uri;
    }

# Settings for a TLS enabled server.
#
    server {
        listen       443 ssl http2 default_server;
        listen       [::]:443 ssl http2 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        ssl_certificate "/root/cert.crt";
        ssl_certificate_key "/root/key.key";
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }
               
        location /auth/ {
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_pass http://localhost:8181/auth/;

        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

}

[root@Fingon nginx]#

Now the UI looks like below:




You can access keycloak on https only using the URL https://fingon/auth.
Any requests to http will be redirected to https.
On Nginx, we need to set the host, x-forwarded-for and x-forwarded-proto headers so that keycloak identifies it is working behind a reverse proxy and does a proper redirection.
More information can be found in keycloak documentation.

Now we are ready to do more keycloak configurations ☺

Python & Redash data regression modeling

I was trying to do some performance data curve fitting on data related to a Java application heap size scaling against number of data objects used by application, and I thought to test the possibility of using python NumPy library to do some curve fitting using polynomial regression also thought to explore capabilities of Redash as a dash boarding tool able to pull the data from JDBC datasources and offers SQL query interface to obtain the data used for graphs from backend database.

Python proved to be quite handy and offers better regression capabilities.
Also used the Matplotlib library to do the curve plotting.
This is the sample code: https://github.com/sherif-abdelfattah/Pyplots_regression
And below is the output:

I also experimented with Postgres built-in aggregate functions that does slope and intercept calculation using least squares method, and plotted the data into Redash.
This did work too, though Redash didn't offer a decent way of doing curve fitting and I had to rely on doing the calculation work on Postgres side, mainly creating extra tables and views to hold the Linear regression information to be displayed on Redash dashboards.
On Postgres side, I created views similar to the one below:

create view reg_obj_heap as
select (SELECT regr_intercept("objcount", "max_heap") intercept FROM obj_heap) + (SELECT regr_slope("objcount", "max_heap") as slope FROM obj_heap) * "max_heap"
 as reg_obj, "max_heap" from obj_heap; 


On Redash side, I used another SQL query to build the graph:

WITH group1 AS (
  SELECT "objcount", "max_heap",
  case
 when
    "max_heap" >999 and "max_heap" <999999 then "max_heap"/1000||  ' kb'
 when
    "max_heap" >999999 and "max_heap" <999999999 then "max_heap"/1000000||  ' mb'
 when
    "max_heap" >999999999 then "max_heap"/1000000000||  ' gb'
end as "Heap"
    FROM obj_heap
),
group2 AS (
  SELECT reg_obj, "max_heap"
    FROM reg_obj_heap
)
SELECT Distinct group1.*,group2.reg_obj
  FROM group1
  JOIN group2 ON group1."max_heap" = group2."max_heap"
  order by "max_heap"; 

Complicated SQL grouping is needed to allow drawing 2 series on the Redash graph, the graph looks like this:

Python on the other hand offers good regression capabilities and still can do some decent graphics similar to what could be done in Redash, you can even export the Matplotlib graphs to html format using the library mpld3 and build your own dashboards.

The use of both Python and Redash for data analytics should help me to do effective capacity planning and performance modeling for my services in the future.

Testing http/2 on Apache and Centos 8

Back in 2015 Apache (2.4.17 and onwards) started to support http/2 protocol through a dedicated apache module mod_http2.
CentOS 8  ships with Apache/2.4.37 and thus is cabable of serving http/2 and it was worth a test to try to see how this innovative protocol update works.

Apache on CentOS 8 ships with mod_http2 enabled by default, but in order to use the http/2 protocol, one needs to specify it expelicity using Protocols directive. (Note that it is Protocols with an 's').
Below is a sample Apache config that enables both http/2 on plain/clear text 'h2c' and standard h2 which works on top of SSL using SSL ALPN(Application Layer Protocol Negociation).
Please note that SSL must be enabled thus Openssl should be installed and Apache mod_ssl should also be installed and an https should be configured for h2 to work.


Apache config:

[root@beren ~]# cat /etc/httpd/conf.d/http2link.conf
### Adding http2 link headers and  H2PushResource
Protocols h2c h2 http/1.1
H2EarlyHints on

Header add Link "</test/ysf_100.png>; rel=preload; as=image"
Header add Link "</test/ysf_099.png>; rel=preload; as=image"
Header add Link "</test/ysf_098.png>; rel=preload; as=image"
Header add Link "</test/ysf_097.png>; rel=preload; as=image"
Header add Link "</test/ysf_096.png>; rel=preload; as=image"

H2PushResource /test/ysf.png
H2PushResource /test/ysf_096.png
H2PushResource /test/ysf_095.png
H2PushResource /test/ysf_094.png
H2PushResource /test/ysf_093.png
H2PushResource /test/ysf_092.png

[root@beren ~]#

The configuration mainly contains the Protocols directive which lists the prefered protocls Apache will offer to the client starting with prefered from left to right, ordering matters as per Apache documentation.

Then we expelicty set early hints to on, this feature will make use of the http/2 server push features to speed up page load times.
There are 2 ways Apache can use the http/2 server push, either by adding the 'Link' header as shown above using mod_header 'Header add' directive or using the new mod_http2 'H2PushResource' directive to push a certain resource to the client using early hints.
I have created an HTML page with some 110 image resources and tried to push the above subset of resources to test the configuration.

I tried 3 different clients, curl (curl now supports http/2 if compiled with the nghttp2 library, CentOS 8 offer curl compiled with http/2 feature), nghttp client tool and Vivaldi browser (Chromium like).

Below is the output of curl in connecting to Apache using h2c (http/2 on clear text), the protocol upgrade is visible and you can see indeed there is an extra reponse with HTTP 101 Protocol upgrade before the standard HTTP 200 response:

sherif@fingolfin:~$ curl -v --http2 http://192.168.56.105/test/test.html >/dev/null
*   Trying 192.168.56.105...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.56.105 (192.168.56.105) port 80 (#0)
> GET /test/test.html HTTP/1.1
> Host: 192.168.56.105
> User-Agent: curl/7.58.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
>
< HTTP/1.1 101 Switching Protocols
< Upgrade: h2c
< Connection: Upgrade
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 103
< link: </test/ysf.png>; rel=preload, </test/ysf_096.png>; rel=preload, </test/ysf_095.png>; rel=preload, </test/ysf_094.png>; rel=preload, </test/ysf_093.png>; rel=preload, </test/ysf_092.png>; rel=preload
< HTTP/2 200
< date: Sun, 00 Jan 1900 00:00:00 GMT
< server: Apache/2.4.37 (centos) OpenSSL/1.1.1c
< last-modified: Fri, 17 Apr 2020 18:57:51 GMT
< etag: W/"1326-5a3812047b27b"
< accept-ranges: bytes
< content-length: 4902
< link: </test/ysf_100.png>; rel=preload; as=image
< link: </test/ysf_099.png>; rel=preload; as=image
< link: </test/ysf_098.png>; rel=preload; as=image
< link: </test/ysf_097.png>; rel=preload; as=image
< link: </test/ysf_096.png>; rel=preload; as=image
< content-type: text/html; charset=UTF-8
<
{ [4902 bytes data]
100  4902  100  4902    0     0  2393k      0 --:--:-- --:--:-- --:--:-- 2393k
* Connection #0 to host 192.168.56.105 left intact
sherif@fingolfin:~$ 

You can also see the Link headers added to the response from Apache with preload, this should direct the browser to receive those resoruces early on to speed up page loading.
Then below is the output of using http/2 over SSL, this time using h2 with SSL ALPN negociation, Apache is seen offering both http/2 then http/1.1 using ALPN and using TLS1.3 during the handshake:

sherif@fingolfin:~$ curl -v -k --http2 https://192.168.56.105/test/test.html >/dev/null
*   Trying 192.168.56.105...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 192.168.56.105 (192.168.56.105) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Unknown (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2672 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; O=Unspecified; CN=beren; emailAddress=root@beren
*  start date: Apr 16 19:10:23 2020 GMT
*  expire date: Apr 21 20:50:23 2021 GMT
*  issuer: C=US; O=Unspecified; OU=ca-5235634170413358813; CN=beren; emailAddress=root@beren
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* Using Stream ID: 1 (easy handle 0x55e221b8d580)
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
> GET /test/test.html HTTP/2
> Host: 192.168.56.105
> User-Agent: curl/7.58.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
< HTTP/2 103
< link: </test/ysf.png>; rel=preload, </test/ysf_096.png>; rel=preload, </test/ysf_095.png>; rel=preload, </test/ysf_094.png>; rel=preload, </test/ysf_093.png>; rel=preload, </test/ysf_092.png>; rel=preload
{ [5 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
< HTTP/2 200
< date: Fri, 17 Apr 2020 19:59:34 GMT
< server: Apache/2.4.37 (centos) OpenSSL/1.1.1c
< last-modified: Fri, 17 Apr 2020 18:57:51 GMT
< etag: "1326-5a3812047b27b"
< accept-ranges: bytes
< content-length: 4902
< link: </test/ysf_100.png>; rel=preload; as=image
< link: </test/ysf_099.png>; rel=preload; as=image
< link: </test/ysf_098.png>; rel=preload; as=image
< link: </test/ysf_097.png>; rel=preload; as=image
< link: </test/ysf_096.png>; rel=preload; as=image
< content-type: text/html; charset=UTF-8
<
{ [978 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
100  4902  100  4902    0     0  76593      0 --:--:-- --:--:-- --:--:-- 76593
* Connection #0 to host 192.168.56.105 left intact
sherif@fingolfin:~$

The next test is done with the nghttp client tool that ships with nghttp2/1.33.0 and is a client implementation for the http/2 C liberary libnghttp2.
The main objective here is to test resoruces being pushed and seening if it works as expected and the differance between resources pushed with early hints and resources pushed with Link header:

Testing pushes:

sherif@fingolfin:~$ nghttp -vnys https://192.168.56.105/test/test.html
[  0.003] Connected
The negotiated protocol: h2
[  0.054] recv SETTINGS frame <length=6, flags=0x00, stream_id=0>
          (niv=1)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
[  0.054] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=2147418112)
[  0.054] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.054] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.054] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.054] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.054] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.054] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.054] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.054] send HEADERS frame <length=50, flags=0x25, stream_id=13>
          ; END_STREAM | END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: GET
          :path: /test/test.html
          :scheme: https
          :authority: 192.168.56.105
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.30.0
[  0.056] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.057] recv (stream_id=13) :scheme: https
[  0.057] recv (stream_id=13) :authority: 192.168.56.105
[  0.057] recv (stream_id=13) :path: /test/ysf.png
[  0.057] recv (stream_id=13) :method: GET
[  0.057] recv (stream_id=13) accept: */*
[  0.057] recv (stream_id=13) accept-encoding: gzip, deflate
[  0.057] recv (stream_id=13) user-agent: nghttp2/1.30.0
[  0.057] recv (stream_id=13) host: 192.168.56.105
[  0.057] recv PUSH_PROMISE frame <length=60, flags=0x04, stream_id=13>
          ; END_HEADERS
......          (padlen=0)
          ; First push response header
[  0.067] recv (stream_id=20) :status: 200
[  0.067] recv (stream_id=20) date: Fri, 17 Apr 2020 20:02:08 GMT
[  0.067] recv (stream_id=20) server: Apache/2.4.37 (centos) OpenSSL/1.1.1c
[  0.067] recv (stream_id=20) last-modified: Thu, 16 Apr 2020 20:38:41 GMT
[  0.067] recv (stream_id=20) etag: "36ec-5a36e6b16758e"
[  0.067] recv (stream_id=20) accept-ranges: bytes
[  0.067] recv (stream_id=20) content-length: 14060
[  0.067] recv (stream_id=20) link: </test/ysf_100.png>; rel=preload; as=image
[  0.067] recv (stream_id=20) link: </test/ysf_099.png>; rel=preload; as=image
[  0.067] recv (stream_id=20) link: </test/ysf_098.png>; rel=preload; as=image
[  0.067] recv (stream_id=20) link: </test/ysf_097.png>; rel=preload; as=image
[  0.067] recv (stream_id=20) link: </test/ysf_096.png>; rel=preload; as=image
[  0.067] recv (stream_id=20) content-type: image/png
[  0.067] recv HEADERS frame <length=37, flags=0x04, stream_id=20>
          ; END_HEADERS
          (padlen=0)
[  0.067] recv (stream_id=2) :status: 103
[  0.067] recv (stream_id=2) link: </test/ysf.png>; rel=preload, </test/ysf_096.png>; rel=preload, </test/ysf_095.png>; rel=preload, </test/ysf_094.png>; rel=preload, </test/ysf_093.png>; rel=preload, </test/ysf_092.png>; rel=preload
[  0.067] recv HEADERS frame <length=2, flags=0x04, stream_id=2>
          ; END_HEADERS
          (padlen=0)
          ; First push response header
[  0.068] recv (stream_id=2) :status: 200
[  0.068] recv (stream_id=2) date: Fri, 17 Apr 2020 20:02:08 GMT
[  0.068] recv (stream_id=2) server: Apache/2.4.37 (centos) OpenSSL/1.1.1c
[  0.068] recv (stream_id=2) last-modified: Thu, 16 Apr 2020 20:37:22 GMT
[  0.068] recv (stream_id=2) etag: "36ec-5a36e665c587d"
........[  0.079] recv DATA frame <length=1291, flags=0x00, stream_id=20>
[  0.079] recv DATA frame <length=487, flags=0x01, stream_id=14>
          ; END_STREAM
[  0.079] recv DATA frame <length=1291, flags=0x00, stream_id=2>
[  0.079] recv DATA frame <length=1150, flags=0x01, stream_id=20>
          ; END_STREAM
[  0.079] recv DATA frame <length=167, flags=0x01, stream_id=2>
          ; END_STREAM
[  0.079] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=20, error_code=NO_ERROR(0x00), opaque_data(0)=[])
***** Statistics *****

Request timing:
  responseEnd: the  time  when  last  byte of  response  was  received
               relative to connectEnd
 requestStart: the time  just before  first byte  of request  was sent
               relative  to connectEnd.   If  '*' is  shown, this  was
               pushed by server.
      process: responseEnd - requestStart
         code: HTTP status code
         size: number  of  bytes  received as  response  body  without
               inflation.
          URI: request URI

see http://www.w3.org/TR/resource-timing/#processing-model

sorted by 'complete'

id  responseEnd requestStart  process code size request path
 13     +9.49ms       +675us   8.81ms  200   4K /test/test.html
  4    +13.76ms *    +3.93ms   9.83ms  200  13K /test/ysf_096.png
  6    +23.55ms *    +4.11ms  19.44ms  200  13K /test/ysf_095.png
  8    +23.60ms *    +4.29ms  19.31ms  200  13K /test/ysf_094.png
 10    +23.65ms *    +4.68ms  18.96ms  200  13K /test/ysf_093.png
 12    +23.93ms *    +6.61ms  17.32ms  200  13K /test/ysf_092.png
 16    +25.72ms *    +8.12ms  17.59ms  200  13K /test/ysf_099.png
 18    +25.75ms *    +8.29ms  17.46ms  200  13K /test/ysf_098.png
 14    +25.81ms *    +7.96ms  17.85ms  200  13K /test/ysf_100.png
 20    +25.87ms *    +8.58ms  17.28ms  200  13K /test/ysf_097.png
  2    +25.90ms *    +3.51ms  22.39ms  200  13K /test/ysf.png
sherif@fingolfin:~$ 

From above output, we can see all the resources that are starred are pushed by the server, both sets of resources pushed with 'Link' header and using the 'H2PushResource' are visible.
The resource ysf_096.png was pushed once, even though it was mentioned twice in the config, also resources pushed with early hint mechanism using 'H2PushResource' are sent using the HTTP 103 Early Hints response containing the Link header which could also be seen in the previous curl output.
Last test is to verify how browsers are handling server pushes.
This was done using Vivldi browser:

Using http/2 is gaining increasing popularity specially with CDN networks as it helps cutting down page load times by a big margin.
Dynamic applications using languages like php, Java and the like can always inject Link headers in their response to push large resources that might be still referneced deep inside the page or one of its dependancies to speed up page load time.