Update Haproxy configuration using Data Plane API and REST

Update Haproxy configuration using Data Plane API and REST / 24 Nov 2019 / Author: Haim Ari

Estimated read time: 12 minutes

Motivation

Updating haproxy configuration can be made using a REST request (curl/python script etc…) instead of updating file. One of the tasks i had in order to migrate a service from the cloud to one of the “On Premise” Data-centers was to make sure it is possible to quickly update all Haproxy nodes, and reload them with the new config. I didn’t want to create another config file, parse the existing one, or do anything related to config files.

In the previous post “Create an haproxy cluster running with docker” I wrote how to set up haproxy on docker.

In this post (with some modifications) we will:

Create a docker image containing the data-plane-api binary file.
Enable the data-plane-api as a program in our haproxy.conf.
Call the data-plane-api REST API, in order to modify and reload configuration (replace the backend servers PORT)

What is data-plane-api ?

API for editing and managing haproxy instances. Provides process information, configuration management, haproxy stats and logs.

You can read about it Here

In general this is the binary which runs and exposes A REST API while communicating with haproxy

Build the data-plane-api docker image:

We will build a docker image so we could later on COPY the resulting binary file of that image into our haproxy image:

FROM golang:1.13.4-alpine3.10

RUN apk add --update \
    git \
    make \
    bash \
  && rm -rf /var/cache/apk/*

RUN git clone https://github.com/haproxytech/dataplaneapi.git && \
    cd dataplaneapi && \
    make build
RUN pwd
RUN dataplaneapi/build/dataplaneapi -v
WORKDIR dataplaneapi/build/

CMD ["./dataplaneapi"]

Build, tag and push this image to your registry (It can be public as we have no code in this image) Or just pull my image:

docker pull imneo1/haproxy-dataplane-api:v1.2.4

Build our haproxy image

Again, you can read the previous post about running haproxy on docker to get more details on that

ARG VERSION

# The Above is the requested haproxy version.
FROM haproxy:${VERSION}-alpine

# This the artifact containing all SSL Certs, created from the gitlab ci.
ENV SSL_AFACTS ssl.tgz

# Extract the SSL archive into the Haproxy ssl directory
ADD $SSL_AFACTS /

RUN mkdir -p /run/haproxy/

# Copy Haproxy confiuration file, This is only required to set the

# Test haproxy configuration
# RUN haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg

Build your haproxy as you normally would, just add the following line which adds the resulting binary from the data-plane-api image.

COPY --from=imneo1/haproxy-dataplane-api:v1.2.4 /go/dataplaneapi/build/dataplaneapi /usr/sbin/dataplaneapi

# Print haproxy information

RUN haproxy -vv
RUN /usr/sbin/dataplaneapi -v

So now you have the data plane api binary file available inside your haproxy image.

Run the dataplaneapi & connect it to haproxy

For this to work make sure you run the haproxy container with:
```
  volumes:
    - /etc/haproxy/:/usr/local/etc/haproxy/:rw
    - /var/lib/haproxy:/var/lib/haproxy:rw
```
In your haproxy.cfg add this configuration to the ‘global’ section
```
global
master-worker
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
daemon
stats socket /run/haproxy/admin.sock mode 660 level admin
```
Then add credentials for the data-plane-api to manage haproxy

you should set the PASSWORD Variable in you runtime, this example runs ansible with jinja template

userlist api
    user dataplaneapi insecure-password 

Let Haproxy start and manage the Data Plane API when it starts. Also instruct it not to reload the data-plane-api each time it reloads

program dataplane-api
    command /usr/sbin/dataplaneapi --host 0.0.0.0 --port 9999 --haproxy-bin /usr/local/sbin/haproxy --config-file /usr/local/etc/haproxy/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --reload-delay 5 --userlist api
    no option start-on-reload

Notice we specify how the Data Plane API should run:

Bind to all interfaces (change that to what you need)
Listen on port 9999
The path to haproxy binary
The path to haproxy config file
How to reload haproxy (Without restarting the container)
Time to delay between reloads
The user to use when authenticating the REST api

Now in my case i needed to change the ports of all existing backend servers This is done by using “Transaction”. it’s not a must but it’s the right way in this case.

You “commit all the changes” And then apply them as one transaction.

To do that you must first detect the current transaction-id of the current version

You can read more about transactions Here

The API documentation is Here

So i needed a single script which will:

Get the current version
Get that version transaction id
Retrieve the current servers in the backend
Replace each backend server configuration (change port).
Apply the transaction.

Here is a short bash script to do all of that using curl.

~~I’ll probably migrate this to python but this works too.~~

And below same concept in python:

#!/bin/bash

PORT=$1
PASSWORD=$2

## Get the current version
echo "Getting current configuration version"

VERSION=`curl -s -X GET --user dataplaneapi:"${PASSWORD}" \
"http://localhost:9999/v2/services/haproxy/configuration/frontends" | jq ._version`

echo "Current Version is: $VERSION"

## Create a new Transaction ID
echo "Creating a new Transaction ID..."

T_ID=`curl -s -X POST --user dataplaneapi:"${PASSWORD}" \
-H "Content-Type: application/json" \
http://localhost:9999/v2/services/haproxy/transactions?version="${VERSION}" | jq '.id'| tr -d '"'`

echo "Transaction ID is: $T_ID"

BACK=`curl -s -X GET --user dataplaneapi:"${PASSWORD}" \
 -H "Content-Type: application/json"  \
 "http://localhost:9999/v2/services/haproxy/configuration/servers?backend=backendname" \
 | jq '.data[].name' | tr -d '"'`

echo "Updating Servers Ports"
for SERVER in ${BACK}
do
	echo "${SERVER}"

  BACKEND_SERVER_CONF='{
  "address": "",
  "check": "enabled",
  "max-connections": 100000,
  "name": "",
  "port": "",
  "weight": 200,
  "maxconn": 2000
  }'
  
  if [[ $SERVER =~ 'stg-' ]]; then
  data=$(echo "$BACKEND_SERVER_CONF" | jq ". + {name: \"$SERVER\"} + {address: \"i-$SERVER\"} + {port: $PORT} + {weight: 10}")
  else
  data=$(echo "$BACKEND_SERVER_CONF" | jq ". + {name: \"$SERVER\"} + {address: \"i-$SERVER\"} + {port: $PORT}")
  fi
  curl -s -X PUT --user dataplaneapi:"${PASSWORD}" \
  -H "Content-Type: application/json" \
  -d "$data" \
  "http://localhost:9999/v2/services/haproxy/configuration/servers/$SERVER?backend=backendname&transaction_id=$T_ID"

done

echo "Submitting Transaction"
curl -X PUT --user dataplaneapi:"${PASSWORD}" \
-H "Content-Type: application/json" \
"http://localhost:9999/v2/services/haproxy/transactions/$T_ID"

echo "Done."

And in python

#!/usr/bin/python
import logging
import urllib2
import base64
import json
import sys

__author__ = "Haim Ari"
__license__ = "GPL"
__version__ = "0.0.1"

logger = logging.getLogger('logger: ')
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler(sys.__stdout__)
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(lineno)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)

base_url = "http://localhost:9999/v2/services/haproxy"
config_url = "/configuration"
version_url = base_url + config_url + "/frontends"
transaction_url = base_url + "/transactions?version="
servers_url = base_url + config_url + "/servers?backend=yourbackend"
update_url = base_url + config_url + "/servers/"
submit_url = base_url + "/transactions/"

your_backend_server_conf = {
    "address": "",
    "check": "enabled",
    "max-connections": 100000,
    "name": "",
    "port": "",
    "weight": 200,
    "maxconn": 2000
}


def get_current_version():
    logger.info("Getting current version number of haproxy config")
    try:
        request = urllib2.Request(version_url)
        base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        response = urllib2.urlopen(request)

    except urllib2.URLError as err:
        logger.error("Could not get current version: " + str(err))
        sys.exit(1)
    else:
        r_data = json.loads(response.read())
        ver = str(r_data['_version'])
        logger.info("Found Current version: " + ver)
        return ver


def create_transaction_id():
    ver = get_current_version()
    logger.info("Creating new Transaction ID")
    try:
        tr_url = transaction_url + ver
        request = urllib2.Request(tr_url)
        base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        request.add_header('Content-Type', 'application/json')
        request.get_method = lambda: 'POST'
        response = urllib2.urlopen(request)
    except urllib2.URLError as err:
        logger.error("Could not create Transaction ID: " + str(err))
        sys.exit(1)
    else:
        r_data = json.loads(response.read())
        tid = (r_data['id'])
        logger.info("Created Transaction: " + str(tid))
        return tid


def submit_transaction(transaction):
    logger.info("Submitting Transaction ID to apply updated configuration.")
    try:
        request = urllib2.Request(submit_url + transaction)
        base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        request.add_header('Content-Type', 'application/json')
        request.get_method = lambda: 'PUT'
        response = urllib2.urlopen(request)
    except urllib2.URLError as err:
        logger.error("Could not submit the transaction: " + str(err))
        logger.error("changes were not applied.")
        sys.exit(1)
    else:
        result = json.loads(response.read())
        t_status = json.dumps(result['status'])
        if "success" in t_status:
            logger.info("Transaction: " + str(transaction) + " submitted successfully")
        else:
            logger.error("Something went wrong... submit response is: " + str(result))


def update_your_backend(on_off):
    t_id = create_transaction_id()
    logger.info("getting the current backend servers in Backend...")
    try:
        request = urllib2.Request(servers_url)
        base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
        request.add_header("Authorization", "Basic %s" % base64string)
        response = urllib2.urlopen(request)
    except urllib2.URLError as err:
        logger.error("Could not get current servers in Backend: " + str(err))
        sys.exit(1)
    else:
        result = json.loads(response.read())
        result_len = len(result['data'])
        item = 0
        logger.info("updating configuration...")
        while item < result_len:
            your_backend_server_conf["weight"] = weight['prd']
            backend_server = result['data'][item]['name']
            item += 1
            your_backend_server_conf["port"] = port[on_off]
            your_backend_server_conf["name"] = str(backend_server)
            your_backend_server_conf["address"] = "i-" + str(backend_server)
            y = backend_server.encode('utf-8')
            if "stg" in y:
                your_backend_server_conf["weight"] = weight['stg']
            try:
                url = update_url + str(backend_server) + "?backend=yourbackend&transaction_id=" + t_id
                request = urllib2.Request(url)
                base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
                request.add_header("Authorization", "Basic %s" % base64string)
                request.add_header('Content-Type', 'application/json')
                request.get_method = lambda: 'PUT'
                response = urllib2.urlopen(request, json.dumps(your_backend_server_conf))
                logger.info("Committing update to transaction:")
                logger.info(response.read())
            except urllib2.URLError as err:
                logger.error(str(err))
                sys.exit(1)
    logger.info("Committed configuration changes successfully.")
    submit_transaction(transaction=t_id)


if __name__ == "__main__":
    if len(sys.argv) == 3:
        logger.info("Starting...")
        status = sys.argv[1]
        token = sys.argv[2]
        bidder = {'stop': "off",
                  'start': "on"}
        port = {'stop': 80,
                'start': 8480}
        weight = {'prd': 200,
                  'stg': 10}
        update_your_nackend(on_off=status)
        logger.info("Done.")
    else:
        logger.error("Missing Arguments...")
        sys.exit(1)

So the default options are set using the $BACKEND_SERVER_CONF Variable which is a simple json. Then we overwrite what we need. in my case i overwrite the PORT, and in case these are STG servers i also overwrite the wight.

That’s it

Once the Data Plane API is enabled and secured, it is a much better way (and faster) To update and reload haproxy without replacing config files.

Also, if you will start the container with new backend server, or use another data-api-plane script to add/delete them. This script will still work as expected as it will run on your existing version of haproxy config.

However it will be much better to configure the entire Haproxy using the data-plane-api.