Create an haproxy cluster running with docker / 19 May 2019 / Author: Haim Ari

    Estimated read time: 18 minutes

    General Design

    4 nodes, installed with latest Ubuntu OS (currently 18.04.2 LTS) in high availability mode. The cluster should be able to operate normally with N -1 servers at all time. Floating ip addresses and the haproxy container are managed by pacemaker. The cluster ring is managed by corosync.

    The haproxy configuration is generated from a template via ansible. All haproxy nodes are running the same configuration while binding to the same floating ip addresses. The nodes are protected by iptables. All the configuration of the Host OS is made by Ansible TLS 1.3 Is supported

    Automation flow

    We will use Latest stable Ubuntu server (currently 18.04.2 LTS), and apply custom configuration via Ansible AWX playbook.

    Deployment - when updating haproxy configuration (haproxy.cfg.jinja2) Gitlab will call a Tower Job which will generate the haproxy.cfg file on The host (Haproxy Machine).

    reload haproxy via Gitlab pipeline:

    The container mounts the directory /etc/haproxy/ Then it will reload the haproxy process only, without restarting/replacing the container.

    Ansible will run this command on the host:

    docker kill -s HUP haproxy 
    

    At the Ansible Tower Job, we will see that the “run_haproxy” task was skipped, And the “reload_haproxy” changed so if we docker logs the container, we will see the “Reexecuting” Warning instead of seeing the container exiting:

    [WARNING] 082/151502 (1) : Reexecuting Master process
    

    The above is the default gitlab “reload” job which will run automatically on ‘release’.

    It is also possible to manually run the gitlab ‘restart-haproxy-container’ which will pull the latest released image, and restart the container.

    Release new docker image

    Upgrading the Haproxy image is made via manually running the gitlab pipeline “Build”. for example to upgrade the haproxy version, just change the gitlab variable VERSION: in the gitlab-ci.yml

    truly-seamless-reloads-with-haproxy

    Another great feature is the way the “reload” in haproxy works now. instead of forking another process and replacing the previous process, When we send SIGHUP to the container, it will pass the listening file descriptors to the new process so that the connection is never closed. so active sockets do not drop, instead they are migrated to a new process.

    In test of 1M requests, you can see below that reloading the haproxy did not cause even a single connection to drop. While the number of request was ~400K, the haproxy was reloaded. The results are 0 Failed requests which allows us to reload haproxy anytime needed, without the fear of dropping requests.

    ab -r -c 10 -n 1000000 "http://10.10.10.10/"
    This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/
    
    Benchmarking 200.200.200.101 (be patient)
    Completed 100000 requests
    Completed 200000 requests
    Completed 300000 requests
    Completed 400000 requests
    Completed 500000 requests
    Completed 600000 requests
    Completed 700000 requests
    Completed 800000 requests
    Completed 900000 requests
    Completed 1000000 requests
    Finished 1000000 requests
    
    
    Server Software:        nginx/1.15.10
    Server Hostname:        1.1.1.1
    Server Port:            80
    
    Document Path:          /
    Document Length:        612 bytes
    
    Concurrency Level:      10
    Time taken for tests:   183.318 seconds
    Complete requests:      1000000
    Failed requests:        0
    Total transferred:      846000000 bytes
    HTML transferred:       612000000 bytes
    Requests per second:    5454.99 [#/sec] (mean)
    Time per request:       1.833 [ms] (mean)
    Time per request:       0.183 [ms] (mean, across all concurrent requests)
    Transfer rate:          4506.76 [Kbytes/sec] received
    
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.1      0       2
    Processing:     0    2   2.1      1      64
    Waiting:        0    2   2.1      1      63
    Total:          0    2   2.1      1      64
    

    After optimization, in stress test with a single backend server it handled 1M requests in ~9 seconds with ~110K/sec (The backend server was a dummy nginx)

    Server Software:        nginx/1.15.6
    Server Hostname:        200.200.200.101
    Server Port:            1111
    
    Document Path:          /
    Document Length:        0 bytes
    
    Concurrency Level:      130
    Time taken for tests:   8.772 seconds
    Complete requests:      1000000
    Failed requests:        0
    Keep-Alive requests:    994172
    Total transferred:      160970860 bytes
    HTML transferred:       0 bytes
    Requests per second:    113993.47 [#/sec] (mean)
    Time per request:       1.140 [ms] (mean)
    Time per request:       0.009 [ms] (mean, across all concurrent requests)
    Transfer rate:          17919.56 [Kbytes/sec] received
    
    Connection Times (ms)
                  min  mean[+/-sd] median   max
    Connect:        0    0   0.0      0       2
    Processing:     0    1   0.9      1      31
    Waiting:        0    1   0.9      1      31
    Total:          0    1   0.9      1      31
    
    Percentage of the requests served within a certain time (ms)
      50%      1
      66%      1
      75%      1
      80%      1
      90%      2
      95%      2
      98%      3
      99%      4
     100%     31 (longest request)
    

    more information about “haproxy seamless reloads”: truly-seamless-reloads-with-haproxy

    Scalability

    It would be posiible to easily add a node to the cluster, and obtain all configuration of the Host from Ansible and all the cluster + haproxy configuration, By joining the cluster.

    docker run -p 9101:9101 quay.io/prometheus/haproxy-exporter:v0.9.0  
    --haproxy.scrape-uri="http://user:[email protected]:1111/haproxy?stats;csv"
    

    Scrape Exporter

    curl localhost:9101/metrics
    

    will return haproxy stats with all information including backends

    Tests

    The haproxy.cfg is tested using simple “-c”.

    haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
    

    pacemaker & corosync tests:

    Initial state:

    Cluster name: micro-services-cluster
    Stack: corosync
    Current DC: haproxy4 (4) (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
    Last updated: Mon Apr  1 14:25:28 2019
    Last change: Mon Apr  1 14:10:21 2019 by root via cibadmin on haproxy1
    
    4 nodes configured
    12 resources configured
    
    Online: [ haproxy1 (1) haproxy2 (2) haproxy3 (3) haproxy4 (4) ]
    
    Full list of resources:
    
     ms-200.200.200.21	(ocf::heartbeat:IPaddr2):	Started haproxy1
     ms-200.200.200.22	(ocf::heartbeat:IPaddr2):	Started haproxy3
     ms-200.200.200.23	(ocf::heartbeat:IPaddr2):	Started haproxy4
     ms-200.200.200.24	(ocf::heartbeat:IPaddr2):	Started haproxy2
     Clone Set: haproxy-clone [haproxy]
         haproxy	(ocf::heartbeat:docker):	Started haproxy1
         haproxy	(ocf::heartbeat:docker):	Started haproxy3
         haproxy	(ocf::heartbeat:docker):	Started haproxy2
         haproxy	(ocf::heartbeat:docker):	Started haproxy4
         Started: [ haproxy1 haproxy2 haproxy3 haproxy4 ]
     Clone Set: pingTest-clone [pingTest]
         pingTest	(ocf::pacemaker:ping):	Started haproxy1
         pingTest	(ocf::pacemaker:ping):	Started haproxy3
         pingTest	(ocf::pacemaker:ping):	Started haproxy2
         pingTest	(ocf::pacemaker:ping):	Started haproxy4
         Started: [ haproxy1 haproxy2 haproxy3 haproxy4 ]
    
    Node Attributes:
    * Node haproxy1 (1):
        + pingd                           	: 1000
    * Node haproxy2 (2):
        + pingd                           	: 1000
    * Node haproxy3 (3):
        + pingd                           	: 1000
    * Node haproxy4 (4):
        + pingd                           	: 1000
    
    Migration Summary:
    * Node haproxy1 (1):
    * Node haproxy3 (3):
    * Node haproxy2 (2):
    * Node haproxy4 (4):
    
    PCSD Status:
      haproxy1: Online
      haproxy3: Online
      haproxy4: Online
      haproxy2: Online
    
    Daemon Status:
      corosync: active/disabled
      pacemaker: active/disabled
      pcsd: active/enabled
    

    Security

    The Haproxy hosts network is protected by iptables: Added a custom module named iptables_raw

    iptables_raw details:

    • The ansible module source on github

    • The module Documentation

    • SNYK integration, Scanning base image and Dockerfile.
    • SSL certs, and stats page authentication are configured using gitlab SECRETS.
    • The SSL certs are part of the Docker image and a ‘release’ is required to deploy them.

    TLS 1.3 is supported

    HA-Proxy version 1.9.4 2019/02/06 - https://haproxy.org/
    Build options :
      TARGET  = linux2628
      CPU     = generic
      CC      = gcc
      CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
      OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
    
    Default settings :
      maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
    
    Built with OpenSSL version : OpenSSL 1.1.1a  20 Nov 2018
    Running on OpenSSL version : OpenSSL 1.1.1a  20 Nov 2018
    OpenSSL library supports TLS extensions : yes
    OpenSSL library supports SNI : yes
    OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
    

    Kernel Optimization:

    These cpu bindings of IRQ are made manually since nothing promises, The interface name, IRQ Number or how the kernel maps the cores of more than 1 CPU

    The container is started in privileged mode and specific kernel parameters. They are set up at the run_haproxy role which starts the container. (The results can be shown with docker inspect for example)

    docker inspect haproxy
    

    Output:

    "Sysctls": {
                    "net.core.netdev_max_backlog": "100000",
                    "net.core.somaxconn": "65534",
                    "net.ipv4.conf.all.send_redirects": "1",
                    "net.ipv4.ip_local_port_range": "1025 65000",
                    "net.ipv4.ip_nonlocal_bind": "1",
                    "net.ipv4.tcp_abort_on_overflow": "0",
                    "net.ipv4.tcp_fin_timeout": "10",
                    "net.ipv4.tcp_keepalive_time": "300",
                    "net.ipv4.tcp_max_orphans": "262144",
                    "net.ipv4.tcp_max_syn_backlog": "100000",
                    "net.ipv4.tcp_max_tw_buckets": "262144",
                    "net.ipv4.tcp_reordering": "3",
                    "net.ipv4.tcp_rmem": "4096    87380    4120928",
                    "net.ipv4.tcp_syn_retries": "5",
                    "net.ipv4.tcp_synack_retries": "3",
                    "net.ipv4.tcp_syncookies": "1",
                    "net.ipv4.tcp_timestamps": "0",
                    "net.ipv4.tcp_tw_reuse": "1",
                    "net.ipv4.tcp_wmem": "4096 16384 262144",
                    "net.netfilter.nf_conntrack_max": "10485760",
                    "net.netfilter.nf_conntrack_tcp_timeout_fin_wait": "30",
                    "net.netfilter.nf_conntrack_tcp_timeout_time_wait": "15"
                },
    
    

    Availability

    In order to run the Haproxy in High availability & Active-Active mode The following components are configured to create a cluster:

    • pacemaker
    • corosync

    In order to view/update resources configuration such as Floating ip addresses a “pcs” container is running on each node of the cluster

    docker image: pschiffe/pcs

    Running the container manually (not needed as it runs via ansible):

    docker run -d --privileged --net=host \
        -v /etc/haproxy/backup/pcs:/etc/haproxy/backup/pcs:rw  \
        -v /sys/fs/cgroup:/sys/fs/cgroup \
        -v /etc/localtime:/etc/localtime:ro \
        -v /run/docker.sock:/etc/docker.sock -v  \
        /usr/bin/docker:/usr/bin/docker:ro --name pcs \ 
        pschiffe/pcs
    

    The ansible role run_pcs will run the pcs container with the relevant docker options:

    - name: Run PCS container
      docker_container:
        recreate: no
        pull: true
        name: pcs
        image: pschiffe/pcs
        network_mode: host
        state: started
        restart: no
        stop_signal: SIGUSR1
        privileged: yes
        ports:
         - 1111:1111
        dns_search_domains:
         - "your.domain"
        volumes:
         - /usr/bin/docker:/usr/bin/docker:ro
         - /etc/haproxy/backup/pcs:/etc/haproxy/backup/pcs:rw
         - /sys/fs/cgroup:/sys/fs/cgroup
         - /etc/localtime:/etc/localtime:ro
         - /run/docker.sock:/etc/docker.sock
    

    The container is based on Centos, and therefor uses the ‘pcs’ command(RedHat)

    • When adding a new node to the cluster, set the ‘hacluster’ user password 1st (must be identical in all all containers)
    [email protected]:~# docker exec -it pcs bash
    [[email protected] /]# passwd hacluster
    

    Enter the container and run pcs commands to manage/view configuration. pcs docker resource agent manual Here For example, To see the cluster and resources state run:

    pcs status --full
    

    Examples: creating pcs resources

    Creating the cluster

    pcs cluster auth -u hacluster -p "password-here" haproxy1 haproxy2 haproxy3 haproxy4
    pcs cluster setup --name micro-services-cluster haproxy1 haproxy2 haproxy3 haproxy4
    pcs cluster start --all
    

    Create Public Vips

    pcs resource create ms-200.200.200.21 ocf:heartbeat:IPaddr2 ip=200.200.200.21 cidr_netmask=26 op monitor interval=30s
    pcs resource create ms-200.200.200.22 ocf:heartbeat:IPaddr2 ip=200.200.200.22 cidr_netmask=26 op monitor interval=30s
    pcs resource create ms-200.200.200.23 ocf:heartbeat:IPaddr2 ip=200.200.200.23 cidr_netmask=26 op monitor interval=30s
    pcs resource create ms-200.200.200.24 ocf:heartbeat:IPaddr2 ip=200.200.200.24 cidr_netmask=26 op monitor interval=30s
    

    Create the Haproxy resource & clone

    pcs resource create haproxy ocf:heartbeat:docker \
        image=registry.your-domain.com/ansible/playbooks/haproxy \
        allow_pull=1 reuse=1  run_opts='--net=host \
        -v /etc/haproxy:/usr/local/etc/haproxy' \
        mount_points="/etc/haproxy" \
        --group haproxy-group
        
    pcs resource cleanup haproxy-clone
    

    Restoring the entire cluster configuration

    only needed if the entire cluster of the physical nodes or all pcs containers crashed at the same time.

    Backup of the pcs configuration is done automatically once a day by Gitlab scheduler which invokes the tower job with ‘extra variable’: “command=backup”

    backup-pcs:
      image: registry.your-domain.com/public/tower-cli
      stage: backup-pcs
      script:
        - tower-cli config host ${TOWER_HOST}
        - tower-cli config username ${tower_user}
        - tower-cli config password ${tower_password}
        - tower-cli job launch --job-template ${job_id} \
          --extra-vars="command=backup STATS_USER=${STATS_USER} STATS_PASSWORD=${STATS_PASSWORD}" --monitor
      environment:
        name: master
      only:
        - schedules
      dependencies: []
    

    The daily backups are locate on each host under:

    /etc/haproxy/backup/pcs/
    

    manual backup & restore

    Backup:

    pcs config backup backup
    

    Restore:

    pcs config restore backup.tar.bz2
    

    what if a single pcs container crashed or you are adding a new node to the cluster ?

    The 1st stage is: run the ansible playbook run_pcs which will start a fresh container on the node.

    once it is up, follow these steps:

    on the crashed/new host:

    1. Enter the container
    docker exec -it pcs bash
    
    1. Create symlink so it could manage container on the host of itself.
    ln -s /etc/docker.sock run/docker.sock
    
    1. Set Password for hacluster user
    passwd hacluster
    

    Now go to another pre-existing and online pcs container of the cluster

    1. Add authentication for the the new/crashed node and the remaining cluster nodes.
    pcs cluster auth -u hacluster -p "password-here" haproxy1 haproxy2 haproxy3 haproxy4
    
    1. If this is a previously crashed node/pcs container, delete the node before re-adding it.

      otherwise skip to the next step.

    pcs cluster node delete haproxyX
    
    1. Add the node to the cluster:
    pcs cluster setup --name micro-services-cluster haproxy1 haproxy2 haproxy3 haproxy4
    
    1. start the node or the entire cluster for it to take resources.
    pcs cluster start haproxyx
    

    OR

    pcs cluster start --all
    

    This node should now become part of the cluster.

    It is also possible to view pacemaker, corosync and resources status from Web UI, which is available on any online node, for example:

    https://haproxy1:2224

    Gitlab CI:

    image: docker:stable
    
    stages:
      - prepare-ssl
      - build
      - docker-scan
      - release
      - build-release
      - reload-haproxy
      - restart-haproxy-container
      - backup-pcs
    
    
    variables:
      IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
      DEV_TAG: $CI_REGISTRY_IMAGE:dev-$CI_COMMIT_SHA
      VERSION: 1.9.7
      DOCKER_BASE_IMAGE: haproxy:${VERSION}-alpine
      LATEST_DEV: $CI_REGISTRY_IMAGE:${VERSION}-dev
      LATEST: $CI_REGISTRY_IMAGE:${VERSION}-$CI_COMMIT_TAG
      DIR: $CI_PROJECT_DIR
    
    
    prepare-ssl:
      stage: prepare-ssl
      script:
        - mkdir ssl
        - echo "${YOURDOMAIN_COM}" > ssl/yourdomain.com.pem
        - tar -czvf ssl.tgz ssl
      artifacts:
        paths:
          - ssl.tgz
        expire_in: 6hrs
      environment:
        name: master
      tags:
        - generic
      only:
        - web
      except:
        - schedules
    
    
    build-dev:
      stage: build
      script:
        - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
        - docker build --ulimit nofile=10000:10000 --build-arg VERSION=${VERSION} -t $DEV_TAG .
        - docker tag $DEV_TAG $LATEST_DEV
        - docker push $DEV_TAG
        - docker push $LATEST_DEV
      dependencies:
        - prepare-ssl
      except:
        - tags
        - schedules
      only:
        - web
    
    
    .docker-scan:
      stage: docker-scan
      variables:
        MONITOR: "true"
        USER_ID: $GITLAB_USER_LOGIN
      image:
        name: snyk/snyk-cli:docker
        entrypoint:
          - '/usr/bin/env'
          - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
      script:
        - docker pull $DOCKER_BASE_IMAGE
        - snyk test --docker $DOCKER_BASE_IMAGE --file=Dockerfile --project-name=$CI_PROJECT_PATH || true
        - snyk monitor --docker $DOCKER_BASE_IMAGE --file=Dockerfile --project-name=$CI_PROJECT_PATH || true
      except:
        - tags
        - schedules
      only:
        - web
      dependencies: []
    
    
    release:
      stage: release
      image: registry.gitlab.com/juhani/go-semrel-gitlab:v0.20.0
      script:
        - release next-version --allow-current
        - release changelog
        - release commit-and-tag --create-tag-pipeline CHANGELOG.md
      only:
        - master
      except:
        - tags
        - schedules
      when: manual
      environment:
        name: production
      dependencies: []
    
    
    build-release:
      image: docker:latest
      stage: build-release
      script:
        - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
        - docker pull ${LATEST_DEV}
        - docker tag ${LATEST_DEV} ${LATEST}
        - docker tag ${LATEST_DEV} ${CI_REGISTRY_IMAGE}:latest
        - docker push ${LATEST}
        - docker push ${CI_REGISTRY_IMAGE}:latest
      environment:
        name: master
      only:
        - tags
      except:
        - schedules
      dependencies: []
    
    
    reload-haproxy:
      image: registry.yourdomain.com/public/tower-cli
      stage: reload-haproxy
      script:
        - tower-cli config host ${TOWER_HOST}
        - tower-cli config username ${tower_user}
        - tower-cli config password ${tower_password}
        - tower-cli job launch --job-template ${job_id} \
          --extra-vars="STATS_USER=${STATS_USER} \
          STATS_PASSWORD=${STATS_PASSWORD}" --monitor
      environment:
        name: master
      dependencies: []
      when: manual
      except:
        - schedules
    
    
    restart-haproxy-container:
      image: registry.your-domain.com/private/tower-cli
      stage: restart-haproxy-container
      script:
        - tower-cli config host ${TOWER_HOST}
        - tower-cli config username ${tower_user}
        - tower-cli config password ${tower_password}
        - tower-cli job launch --job-template ${job_id} \
          --extra-vars="command=run STATS_USER=${STATS_USER} \
          STATS_PASSWORD=${STATS_PASSWORD}" --monitor
      environment:
        name: master
      only:
        - tags
      tags:
        - generic
      when: manual
      except:
        - schedules
      dependencies: []
    
    backup-pcs:
      image: registry.your-domain.com/private/tower-cli
      stage: backup-pcs
      script:
        - tower-cli config host ${TOWER_HOST}
        - tower-cli config username ${tower_user}
        - tower-cli config password ${tower_password}
        - tower-cli job launch --job-template ${job_id}  \
          --extra-vars="command=backup STATS_USER=${STATS_USER}  \
          STATS_PASSWORD=${STATS_PASSWORD}" --monitor
      environment:
        name: master
      only:
        - schedules
      dependencies: []
    

    Ansible Playbook:

    
    - hosts: haproxy-nj
      roles:
        - role: run_pcs
          when: " 'run' in command or 'reload' in command "
      serial: 1
    
    - hosts: haproxy-nj
      roles:
        - role: download_config
          when: " 'run' in command or 'reload' in command "
      serial: 4
    
    - hosts:
        - haproxy-nj
      roles:
        - role: haproxy_iptables
          when: " 'run' in command "
      serial: 4
    
    - hosts: haproxy-nj
      roles:
        - role: run_haproxy
          when: " 'run' in command "
      serial: 1
    
    - hosts: haproxy-nj
      roles:
        - role: reload_haproxy
          when: " 'reload' in command "
      serial: 1
    
    
    - hosts: haproxy-nj
      roles:
        - role: backup_pcs
          when: " 'backup' in command "
      serial: 1
    

    Dockerfile:

    ARG VERSION
    # The Above is the requested haproxy version.
    FROM haproxy:${VERSION}-alpine
    
    # This the artifact containing all SSL Certs, created from the gitlab ci.
    ENV SSL_AFACTS ssl.tgz
    
    # Extract the SSL archive into the Haproxy ssl directory
    ADD $SSL_AFACTS /
    
    # Print haproxy information
    RUN haproxy -vv
    

    Haproxy configuration (haproxy.cfg.jinja2)

    global
      nbproc 1
      nbthread 38
    #  cpu-map auto:1/1-4 0-3
      cpu-map all 1-39
    
      tune.ssl.default-dh-param  2048
      ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:EECDH+AESGCM:EECDH+CHACHA20
      ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
      ca-base /etc/ssl/certs
      crt-base /etc/ssl/private
      maxconn 100000
    
    defaults
        mode                    http
    {#    option                  httplog#}
        option                  dontlognull
        option http-server-close
        option forwardfor       except 127.0.0.0/8
        option                  redispatch
        option                  abortonclose
        option                  httpclose
        retries                 2
        timeout http-request    10s
        timeout queue           5s
        timeout connect         2s
        timeout client          10s
        timeout client-fin      1s
        timeout server-fin      1s
        timeout server          3s
        timeout http-keep-alive 10s
        timeout check           5s
        maxconn                 2000
    
    
    listen  stats
        bind :2222
        mode            http
    #    log             global
        maxconn 10
        timeout client      100s
        timeout server      100s
        timeout connect     100s
        timeout queue   100s
        stats enable
    # stats hide-version
        stats refresh 30s
        stats show-node
        stats uri  /proxy?stats
        stats auth :
    
    
    frontend mainapi
        no option http-keep-alive
        option httpclose
        maxconn 3000
        mode http
        option  http-use-htx
        bind 200.200.200.21:443 ssl crt /ssl/your-domain.com.pem crt /ssl/your-domain.com.pem
        bind 200.200.200.22:443 ssl crt /ssl/your-domain.com.pem crt /ssl/your-domain.com.pem
        bind 200.200.200.23:443 ssl crt /ssl/your-domain.com.pem crt /ssl/your-domain.com.pem
        bind 200.200.200.24:443 ssl crt /ssl/your-domain.com.pem crt /ssl/your-domain.com.pem
        capture request header User-Agent len 128
        capture request header Referer len 64
        capture request header Host len 64
        capture request header X-Forwarded-For len 64
        capture request header True-Client-IP len 500
    
        default_backend noapi
    
    # Identify Service by URI
        acl is_api path_beg /someapi
    
    # Route to backend by acl
        use_backend some_api if is_some_api
    
    backend some_api
        option http-keep-alive
        no option http-server-close
        no option httpclose
        mode http
        fullconn 10000
        timeout queue 1s
        balance    roundrobin
        option     abortonclose
        option     forwardfor
        timeout server 10s
        option  httpchk GET /some_api/healthcheck
        default-server inter 3s fall 2 rise 3
        http-check expect status 200
        option http-use-htx
        server web-1 200.200.200.158:1111  weight 50 check minconn 100 maxconn 500 maxqueue 1 slowstart 120s
        server web-2 200.200.200.159:1111  weight 50 check minconn 100 maxconn 500 maxqueue 1 slowstart 120s
        server web-2 200.200.200.160:1111  weight 50 check minconn 100 maxconn 500 maxqueue 1 slowstart 120s
    
    backend noapi
        option http-keep-alive
        no option http-server-close
        no option httpclose
        mode http
        fullconn 10000
        timeout queue 1s
        balance    roundrobin
        option     abortonclose
        option     forwardfor
        timeout server 10s
        option  httpchk GET /noapi/healthcheck
        default-server inter 3s fall 2 rise 3
        http-check expect status 200
        option http-use-htx
        server web-1 200.200.200.158:2222  weight 50 check minconn 100 maxconn 500 maxqueue 1 slowstart 120s
        server web-2 200.200.200.159:2222  weight 50 check minconn 100 maxconn 500 maxqueue 1 slowstart 120s
        server web-2 200.200.200.160:2222  weight 50 check minconn 100 maxconn 500 maxqueue 1 slowstart 120s
    

    Ansible Roles:

    Generate haproxy.cfg from the jinja template:

    - name: create /etc/haproxy directory on the Host for docker volume mount
      file:
        path: /etc/haproxy
        state: directory
        mode: 0644
    
    - name: generate haproxy config on the docker Host machine
      vars:
        S_USER: ""
        S_PASS: ""
      template:
        src: haproxy.cfg.jinja2
        dest: /etc/haproxy/haproxy.cfg
    
    

    Run (start/restart haproxy container, some connections will drop)

    - name: Run Haproxy container
      docker_container:
        recreate: yes
        pull: true
        name: haproxy
        image: registry.your-domain.com/ansible/playbooks/haproxy
        network_mode: host
        state: started
        restart: yes
        stop_signal: SIGUSR1
        privileged: yes
        ulimits:
          - nofile:250000:250000
        sysctls:
          net.ipv4.conf.all.rp_filter: 1
          net.core.somaxconn: 65534
          net.core.netdev_max_backlog: 100000
          net.ipv4.ip_local_port_range: 1025 65000
          net.ipv4.conf.all.send_redirects: 1
          net.ipv4.ip_nonlocal_bind: 1
          net.ipv4.tcp_abort_on_overflow: 0
          net.ipv4.tcp_fin_timeout: 10
          net.ipv4.tcp_keepalive_time: 10
          net.ipv4.tcp_max_orphans: 262144
          net.ipv4.tcp_max_syn_backlog: 100000
          net.ipv4.tcp_max_tw_buckets: 262144
          net.ipv4.tcp_rmem: 4096 16060 64060
          net.ipv4.tcp_wmem: 4096 16384 262144
          net.ipv4.tcp_reordering: 3
          net.ipv4.tcp_synack_retries: 3
          net.ipv4.tcp_syncookies: 1
          net.ipv4.tcp_syn_retries: 5
          net.ipv4.tcp_timestamps: 0
          net.ipv4.tcp_tw_reuse: 1
          net.netfilter.nf_conntrack_max: 10485760
          net.netfilter.nf_conntrack_tcp_timeout_fin_wait: 30
          net.netfilter.nf_conntrack_tcp_timeout_time_wait: 15
        ports:
         - 1111:1111
         - 2222:2222
        dns_search_domains:
         - "your-domain.com"
        volumes:
          - /etc/haproxy/:/usr/local/etc/haproxy/:ro
    
    

    Back To Top