Update Haproxy configuration using Data Plane API and REST / 24 Nov 2019 / Author: Haim Ari

    Estimated read time: 12 minutes

    Motivation

    Updating haproxy configuration can be made using a REST request (curl/python script etc…) instead of updating file. One of the tasks i had in order to migrate a service from the cloud to one of the “On Premise” Data-centers was to make sure it is possible to quickly update all Haproxy nodes, and reload them with the new config. I didn’t want to create another config file, parse the existing one, or do anything related to config files.

    In the previous post “Create an haproxy cluster running with docker” I wrote how to set up haproxy on docker.

    In this post (with some modifications) we will:

    • Create a docker image containing the data-plane-api binary file.
    • Enable the data-plane-api as a program in our haproxy.conf.
    • Call the data-plane-api REST API, in order to modify and reload configuration (replace the backend servers PORT)

    What is data-plane-api ?

    API for editing and managing haproxy instances. Provides process information, configuration management, haproxy stats and logs.

    You can read about it Here

    In general this is the binary which runs and exposes A REST API while communicating with haproxy

    Build the data-plane-api docker image:

    We will build a docker image so we could later on COPY the resulting binary file of that image into our haproxy image:

    FROM golang:1.13.4-alpine3.10
    
    RUN apk add --update \
        git \
        make \
        bash \
      && rm -rf /var/cache/apk/*
    
    RUN git clone https://github.com/haproxytech/dataplaneapi.git && \
        cd dataplaneapi && \
        make build
    RUN pwd
    RUN dataplaneapi/build/dataplaneapi -v
    WORKDIR dataplaneapi/build/
    
    CMD ["./dataplaneapi"]
    

    Build, tag and push this image to your registry (It can be public as we have no code in this image) Or just pull my image:

    docker pull imneo1/haproxy-dataplane-api:v1.2.4
    

    Build our haproxy image

    Again, you can read the previous post about running haproxy on docker to get more details on that

    
    ARG VERSION
    
    # The Above is the requested haproxy version.
    FROM haproxy:${VERSION}-alpine
    
    # This the artifact containing all SSL Certs, created from the gitlab ci.
    ENV SSL_AFACTS ssl.tgz
    
    # Extract the SSL archive into the Haproxy ssl directory
    ADD $SSL_AFACTS /
    
    RUN mkdir -p /run/haproxy/
    
    # Copy Haproxy confiuration file, This is only required to set the
    
    # Test haproxy configuration
    # RUN haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
    

    Build your haproxy as you normally would, just add the following line which adds the resulting binary from the data-plane-api image.

    COPY --from=imneo1/haproxy-dataplane-api:v1.2.4 /go/dataplaneapi/build/dataplaneapi /usr/sbin/dataplaneapi
    
    # Print haproxy information
    
    RUN haproxy -vv
    RUN /usr/sbin/dataplaneapi -v
    
    

    So now you have the data plane api binary file available inside your haproxy image.

    Run the dataplaneapi & connect it to haproxy

    • For this to work make sure you run the haproxy container with:
        volumes:
          - /etc/haproxy/:/usr/local/etc/haproxy/:rw
          - /var/lib/haproxy:/var/lib/haproxy:rw
      

      In your haproxy.cfg add this configuration to the ‘global’ section

      global
      master-worker
      chroot /var/lib/haproxy
      pidfile /var/run/haproxy.pid
      daemon
      stats socket /run/haproxy/admin.sock mode 660 level admin
      

      Then add credentials for the data-plane-api to manage haproxy

      you should set the PASSWORD Variable in you runtime, this example runs ansible with jinja template

    userlist api
        user dataplaneapi insecure-password 
    

    Let Haproxy start and manage the Data Plane API when it starts. Also instruct it not to reload the data-plane-api each time it reloads

    program dataplane-api
        command /usr/sbin/dataplaneapi --host 0.0.0.0 --port 9999 --haproxy-bin /usr/local/sbin/haproxy --config-file /usr/local/etc/haproxy/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --reload-delay 5 --userlist api
        no option start-on-reload
    

    Notice we specify how the Data Plane API should run:

    • Bind to all interfaces (change that to what you need)
    • Listen on port 9999
    • The path to haproxy binary
    • The path to haproxy config file
    • How to reload haproxy (Without restarting the container)
    • Time to delay between reloads
    • The user to use when authenticating the REST api

    Now in my case i needed to change the ports of all existing backend servers This is done by using “Transaction”. it’s not a must but it’s the right way in this case.

    You “commit all the changes” And then apply them as one transaction.

    To do that you must first detect the current transaction-id of the current version

    You can read more about transactions Here

    The API documentation is Here

    So i needed a single script which will:

    • Get the current version
    • Get that version transaction id
    • Retrieve the current servers in the backend
    • Replace each backend server configuration (change port).
    • Apply the transaction.

    Here is a short bash script to do all of that using curl.

    I’ll probably migrate this to python but this works too.

    And below same concept in python:

    #!/bin/bash
    
    PORT=$1
    PASSWORD=$2
    
    ## Get the current version
    echo "Getting current configuration version"
    
    VERSION=`curl -s -X GET --user dataplaneapi:"${PASSWORD}" \
    "http://localhost:9999/v2/services/haproxy/configuration/frontends" | jq ._version`
    
    echo "Current Version is: $VERSION"
    
    ## Create a new Transaction ID
    echo "Creating a new Transaction ID..."
    
    T_ID=`curl -s -X POST --user dataplaneapi:"${PASSWORD}" \
    -H "Content-Type: application/json" \
    http://localhost:9999/v2/services/haproxy/transactions?version="${VERSION}" | jq '.id'| tr -d '"'`
    
    echo "Transaction ID is: $T_ID"
    
    BACK=`curl -s -X GET --user dataplaneapi:"${PASSWORD}" \
     -H "Content-Type: application/json"  \
     "http://localhost:9999/v2/services/haproxy/configuration/servers?backend=backendname" \
     | jq '.data[].name' | tr -d '"'`
    
    echo "Updating Servers Ports"
    for SERVER in ${BACK}
    do
    	echo "${SERVER}"
    
      BACKEND_SERVER_CONF='{
      "address": "",
      "check": "enabled",
      "max-connections": 100000,
      "name": "",
      "port": "",
      "weight": 200,
      "maxconn": 2000
      }'
      
      if [[ $SERVER =~ 'stg-' ]]; then
      data=$(echo "$BACKEND_SERVER_CONF" | jq ". + {name: \"$SERVER\"} + {address: \"i-$SERVER\"} + {port: $PORT} + {weight: 10}")
      else
      data=$(echo "$BACKEND_SERVER_CONF" | jq ". + {name: \"$SERVER\"} + {address: \"i-$SERVER\"} + {port: $PORT}")
      fi
      curl -s -X PUT --user dataplaneapi:"${PASSWORD}" \
      -H "Content-Type: application/json" \
      -d "$data" \
      "http://localhost:9999/v2/services/haproxy/configuration/servers/$SERVER?backend=backendname&transaction_id=$T_ID"
    
    done
    
    echo "Submitting Transaction"
    curl -X PUT --user dataplaneapi:"${PASSWORD}" \
    -H "Content-Type: application/json" \
    "http://localhost:9999/v2/services/haproxy/transactions/$T_ID"
    
    echo "Done."
    

    And in python

    #!/usr/bin/python
    import logging
    import urllib2
    import base64
    import json
    import sys
    
    __author__ = "Haim Ari"
    __license__ = "GPL"
    __version__ = "0.0.1"
    
    logger = logging.getLogger('logger: ')
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler(sys.__stdout__)
    ch.setLevel(logging.DEBUG)
    formatter = logging.Formatter('%(asctime)s - %(lineno)s - %(levelname)s - %(message)s')
    ch.setFormatter(formatter)
    logger.addHandler(ch)
    
    base_url = "http://localhost:9999/v2/services/haproxy"
    config_url = "/configuration"
    version_url = base_url + config_url + "/frontends"
    transaction_url = base_url + "/transactions?version="
    servers_url = base_url + config_url + "/servers?backend=yourbackend"
    update_url = base_url + config_url + "/servers/"
    submit_url = base_url + "/transactions/"
    
    your_backend_server_conf = {
        "address": "",
        "check": "enabled",
        "max-connections": 100000,
        "name": "",
        "port": "",
        "weight": 200,
        "maxconn": 2000
    }
    
    
    def get_current_version():
        logger.info("Getting current version number of haproxy config")
        try:
            request = urllib2.Request(version_url)
            base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
            request.add_header("Authorization", "Basic %s" % base64string)
            response = urllib2.urlopen(request)
    
        except urllib2.URLError as err:
            logger.error("Could not get current version: " + str(err))
            sys.exit(1)
        else:
            r_data = json.loads(response.read())
            ver = str(r_data['_version'])
            logger.info("Found Current version: " + ver)
            return ver
    
    
    def create_transaction_id():
        ver = get_current_version()
        logger.info("Creating new Transaction ID")
        try:
            tr_url = transaction_url + ver
            request = urllib2.Request(tr_url)
            base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
            request.add_header("Authorization", "Basic %s" % base64string)
            request.add_header('Content-Type', 'application/json')
            request.get_method = lambda: 'POST'
            response = urllib2.urlopen(request)
        except urllib2.URLError as err:
            logger.error("Could not create Transaction ID: " + str(err))
            sys.exit(1)
        else:
            r_data = json.loads(response.read())
            tid = (r_data['id'])
            logger.info("Created Transaction: " + str(tid))
            return tid
    
    
    def submit_transaction(transaction):
        logger.info("Submitting Transaction ID to apply updated configuration.")
        try:
            request = urllib2.Request(submit_url + transaction)
            base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
            request.add_header("Authorization", "Basic %s" % base64string)
            request.add_header('Content-Type', 'application/json')
            request.get_method = lambda: 'PUT'
            response = urllib2.urlopen(request)
        except urllib2.URLError as err:
            logger.error("Could not submit the transaction: " + str(err))
            logger.error("changes were not applied.")
            sys.exit(1)
        else:
            result = json.loads(response.read())
            t_status = json.dumps(result['status'])
            if "success" in t_status:
                logger.info("Transaction: " + str(transaction) + " submitted successfully")
            else:
                logger.error("Something went wrong... submit response is: " + str(result))
    
    
    def update_your_backend(on_off):
        t_id = create_transaction_id()
        logger.info("getting the current backend servers in Backend...")
        try:
            request = urllib2.Request(servers_url)
            base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
            request.add_header("Authorization", "Basic %s" % base64string)
            response = urllib2.urlopen(request)
        except urllib2.URLError as err:
            logger.error("Could not get current servers in Backend: " + str(err))
            sys.exit(1)
        else:
            result = json.loads(response.read())
            result_len = len(result['data'])
            item = 0
            logger.info("updating configuration...")
            while item < result_len:
                your_backend_server_conf["weight"] = weight['prd']
                backend_server = result['data'][item]['name']
                item += 1
                your_backend_server_conf["port"] = port[on_off]
                your_backend_server_conf["name"] = str(backend_server)
                your_backend_server_conf["address"] = "i-" + str(backend_server)
                y = backend_server.encode('utf-8')
                if "stg" in y:
                    your_backend_server_conf["weight"] = weight['stg']
                try:
                    url = update_url + str(backend_server) + "?backend=yourbackend&transaction_id=" + t_id
                    request = urllib2.Request(url)
                    base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
                    request.add_header("Authorization", "Basic %s" % base64string)
                    request.add_header('Content-Type', 'application/json')
                    request.get_method = lambda: 'PUT'
                    response = urllib2.urlopen(request, json.dumps(your_backend_server_conf))
                    logger.info("Committing update to transaction:")
                    logger.info(response.read())
                except urllib2.URLError as err:
                    logger.error(str(err))
                    sys.exit(1)
        logger.info("Committed configuration changes successfully.")
        submit_transaction(transaction=t_id)
    
    
    if __name__ == "__main__":
        if len(sys.argv) == 3:
            logger.info("Starting...")
            status = sys.argv[1]
            token = sys.argv[2]
            bidder = {'stop': "off",
                      'start': "on"}
            port = {'stop': 80,
                    'start': 8480}
            weight = {'prd': 200,
                      'stg': 10}
            update_your_nackend(on_off=status)
            logger.info("Done.")
        else:
            logger.error("Missing Arguments...")
            sys.exit(1)
    

    So the default options are set using the $BACKEND_SERVER_CONF Variable which is a simple json. Then we overwrite what we need. in my case i overwrite the PORT, and in case these are STG servers i also overwrite the wight.

    That’s it

    Once the Data Plane API is enabled and secured, it is a much better way (and faster) To update and reload haproxy without replacing config files.

    Also, if you will start the container with new backend server, or use another data-api-plane script to add/delete them. This script will still work as expected as it will run on your existing version of haproxy config.

    However it will be much better to configure the entire Haproxy using the data-plane-api.