Update Haproxy configuration using Data Plane API and REST / 24 Nov 2019 / Author: Haim Ari
Estimated read time: 12 minutes
Motivation
Updating haproxy configuration can be made using a REST request (curl/python script etc…) instead of updating file. One of the tasks i had in order to migrate a service from the cloud to one of the “On Premise” Data-centers was to make sure it is possible to quickly update all Haproxy nodes, and reload them with the new config. I didn’t want to create another config file, parse the existing one, or do anything related to config files.
In the previous post “Create an haproxy cluster running with docker” I wrote how to set up haproxy on docker.
In this post (with some modifications) we will:
- Create a docker image containing the data-plane-api binary file.
- Enable the data-plane-api as a program in our haproxy.conf.
- Call the data-plane-api REST API, in order to modify and reload configuration (replace the backend servers PORT)
What is data-plane-api ?
API for editing and managing haproxy instances. Provides process information, configuration management, haproxy stats and logs.
You can read about it Here
In general this is the binary which runs and exposes A REST API while communicating with haproxy
Build the data-plane-api docker image:
We will build a docker image so we could later on COPY the resulting binary file of that image into our haproxy image:
FROM golang:1.13.4-alpine3.10
RUN apk add --update \
git \
make \
bash \
&& rm -rf /var/cache/apk/*
RUN git clone https://github.com/haproxytech/dataplaneapi.git && \
cd dataplaneapi && \
make build
RUN pwd
RUN dataplaneapi/build/dataplaneapi -v
WORKDIR dataplaneapi/build/
CMD ["./dataplaneapi"]
Build, tag and push this image to your registry (It can be public as we have no code in this image) Or just pull my image:
docker pull imneo1/haproxy-dataplane-api:v1.2.4
Build our haproxy image
Again, you can read the previous post about running haproxy on docker to get more details on that
ARG VERSION
# The Above is the requested haproxy version.
FROM haproxy:${VERSION}-alpine
# This the artifact containing all SSL Certs, created from the gitlab ci.
ENV SSL_AFACTS ssl.tgz
# Extract the SSL archive into the Haproxy ssl directory
ADD $SSL_AFACTS /
RUN mkdir -p /run/haproxy/
# Copy Haproxy confiuration file, This is only required to set the
# Test haproxy configuration
# RUN haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
Build your haproxy as you normally would, just add the following line which adds the resulting binary from the data-plane-api image.
COPY --from=imneo1/haproxy-dataplane-api:v1.2.4 /go/dataplaneapi/build/dataplaneapi /usr/sbin/dataplaneapi
# Print haproxy information
RUN haproxy -vv
RUN /usr/sbin/dataplaneapi -v
So now you have the data plane api binary file available inside your haproxy image.
Run the dataplaneapi & connect it to haproxy
- For this to work make sure you run the haproxy container with:
volumes: - /etc/haproxy/:/usr/local/etc/haproxy/:rw - /var/lib/haproxy:/var/lib/haproxy:rw
In your haproxy.cfg add this configuration to the ‘global’ section
global master-worker chroot /var/lib/haproxy pidfile /var/run/haproxy.pid daemon stats socket /run/haproxy/admin.sock mode 660 level admin
Then add credentials for the data-plane-api to manage haproxy
you should set the PASSWORD Variable in you runtime, this example runs ansible with jinja template
userlist api
user dataplaneapi insecure-password
Let Haproxy start and manage the Data Plane API when it starts. Also instruct it not to reload the data-plane-api each time it reloads
program dataplane-api
command /usr/sbin/dataplaneapi --host 0.0.0.0 --port 9999 --haproxy-bin /usr/local/sbin/haproxy --config-file /usr/local/etc/haproxy/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --reload-delay 5 --userlist api
no option start-on-reload
Notice we specify how the Data Plane API should run:
- Bind to all interfaces (change that to what you need)
- Listen on port 9999
- The path to haproxy binary
- The path to haproxy config file
- How to reload haproxy (Without restarting the container)
- Time to delay between reloads
- The user to use when authenticating the REST api
Now in my case i needed to change the ports of all existing backend servers This is done by using “Transaction”. it’s not a must but it’s the right way in this case.
You “commit all the changes” And then apply them as one transaction.
To do that you must first detect the current transaction-id of the current version
You can read more about transactions Here
The API documentation is Here
So i needed a single script which will:
- Get the current version
- Get that version transaction id
- Retrieve the current servers in the backend
- Replace each backend server configuration (change port).
- Apply the transaction.
Here is a short bash script to do all of that using curl.
I’ll probably migrate this to python but this works too.
And below same concept in python:
#!/bin/bash
PORT=$1
PASSWORD=$2
## Get the current version
echo "Getting current configuration version"
VERSION=`curl -s -X GET --user dataplaneapi:"${PASSWORD}" \
"http://localhost:9999/v2/services/haproxy/configuration/frontends" | jq ._version`
echo "Current Version is: $VERSION"
## Create a new Transaction ID
echo "Creating a new Transaction ID..."
T_ID=`curl -s -X POST --user dataplaneapi:"${PASSWORD}" \
-H "Content-Type: application/json" \
http://localhost:9999/v2/services/haproxy/transactions?version="${VERSION}" | jq '.id'| tr -d '"'`
echo "Transaction ID is: $T_ID"
BACK=`curl -s -X GET --user dataplaneapi:"${PASSWORD}" \
-H "Content-Type: application/json" \
"http://localhost:9999/v2/services/haproxy/configuration/servers?backend=backendname" \
| jq '.data[].name' | tr -d '"'`
echo "Updating Servers Ports"
for SERVER in ${BACK}
do
echo "${SERVER}"
BACKEND_SERVER_CONF='{
"address": "",
"check": "enabled",
"max-connections": 100000,
"name": "",
"port": "",
"weight": 200,
"maxconn": 2000
}'
if [[ $SERVER =~ 'stg-' ]]; then
data=$(echo "$BACKEND_SERVER_CONF" | jq ". + {name: \"$SERVER\"} + {address: \"i-$SERVER\"} + {port: $PORT} + {weight: 10}")
else
data=$(echo "$BACKEND_SERVER_CONF" | jq ". + {name: \"$SERVER\"} + {address: \"i-$SERVER\"} + {port: $PORT}")
fi
curl -s -X PUT --user dataplaneapi:"${PASSWORD}" \
-H "Content-Type: application/json" \
-d "$data" \
"http://localhost:9999/v2/services/haproxy/configuration/servers/$SERVER?backend=backendname&transaction_id=$T_ID"
done
echo "Submitting Transaction"
curl -X PUT --user dataplaneapi:"${PASSWORD}" \
-H "Content-Type: application/json" \
"http://localhost:9999/v2/services/haproxy/transactions/$T_ID"
echo "Done."
And in python
#!/usr/bin/python
import logging
import urllib2
import base64
import json
import sys
__author__ = "Haim Ari"
__license__ = "GPL"
__version__ = "0.0.1"
logger = logging.getLogger('logger: ')
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler(sys.__stdout__)
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(lineno)s - %(levelname)s - %(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)
base_url = "http://localhost:9999/v2/services/haproxy"
config_url = "/configuration"
version_url = base_url + config_url + "/frontends"
transaction_url = base_url + "/transactions?version="
servers_url = base_url + config_url + "/servers?backend=yourbackend"
update_url = base_url + config_url + "/servers/"
submit_url = base_url + "/transactions/"
your_backend_server_conf = {
"address": "",
"check": "enabled",
"max-connections": 100000,
"name": "",
"port": "",
"weight": 200,
"maxconn": 2000
}
def get_current_version():
logger.info("Getting current version number of haproxy config")
try:
request = urllib2.Request(version_url)
base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
response = urllib2.urlopen(request)
except urllib2.URLError as err:
logger.error("Could not get current version: " + str(err))
sys.exit(1)
else:
r_data = json.loads(response.read())
ver = str(r_data['_version'])
logger.info("Found Current version: " + ver)
return ver
def create_transaction_id():
ver = get_current_version()
logger.info("Creating new Transaction ID")
try:
tr_url = transaction_url + ver
request = urllib2.Request(tr_url)
base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
request.add_header('Content-Type', 'application/json')
request.get_method = lambda: 'POST'
response = urllib2.urlopen(request)
except urllib2.URLError as err:
logger.error("Could not create Transaction ID: " + str(err))
sys.exit(1)
else:
r_data = json.loads(response.read())
tid = (r_data['id'])
logger.info("Created Transaction: " + str(tid))
return tid
def submit_transaction(transaction):
logger.info("Submitting Transaction ID to apply updated configuration.")
try:
request = urllib2.Request(submit_url + transaction)
base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
request.add_header('Content-Type', 'application/json')
request.get_method = lambda: 'PUT'
response = urllib2.urlopen(request)
except urllib2.URLError as err:
logger.error("Could not submit the transaction: " + str(err))
logger.error("changes were not applied.")
sys.exit(1)
else:
result = json.loads(response.read())
t_status = json.dumps(result['status'])
if "success" in t_status:
logger.info("Transaction: " + str(transaction) + " submitted successfully")
else:
logger.error("Something went wrong... submit response is: " + str(result))
def update_your_backend(on_off):
t_id = create_transaction_id()
logger.info("getting the current backend servers in Backend...")
try:
request = urllib2.Request(servers_url)
base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
response = urllib2.urlopen(request)
except urllib2.URLError as err:
logger.error("Could not get current servers in Backend: " + str(err))
sys.exit(1)
else:
result = json.loads(response.read())
result_len = len(result['data'])
item = 0
logger.info("updating configuration...")
while item < result_len:
your_backend_server_conf["weight"] = weight['prd']
backend_server = result['data'][item]['name']
item += 1
your_backend_server_conf["port"] = port[on_off]
your_backend_server_conf["name"] = str(backend_server)
your_backend_server_conf["address"] = "i-" + str(backend_server)
y = backend_server.encode('utf-8')
if "stg" in y:
your_backend_server_conf["weight"] = weight['stg']
try:
url = update_url + str(backend_server) + "?backend=yourbackend&transaction_id=" + t_id
request = urllib2.Request(url)
base64string = base64.encodestring('%s:%s' % ("dataplaneapi", token)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
request.add_header('Content-Type', 'application/json')
request.get_method = lambda: 'PUT'
response = urllib2.urlopen(request, json.dumps(your_backend_server_conf))
logger.info("Committing update to transaction:")
logger.info(response.read())
except urllib2.URLError as err:
logger.error(str(err))
sys.exit(1)
logger.info("Committed configuration changes successfully.")
submit_transaction(transaction=t_id)
if __name__ == "__main__":
if len(sys.argv) == 3:
logger.info("Starting...")
status = sys.argv[1]
token = sys.argv[2]
bidder = {'stop': "off",
'start': "on"}
port = {'stop': 80,
'start': 8480}
weight = {'prd': 200,
'stg': 10}
update_your_nackend(on_off=status)
logger.info("Done.")
else:
logger.error("Missing Arguments...")
sys.exit(1)
So the default options are set using the $BACKEND_SERVER_CONF Variable which is a simple json. Then we overwrite what we need. in my case i overwrite the PORT, and in case these are STG servers i also overwrite the wight.
That’s it
Once the Data Plane API is enabled and secured, it is a much better way (and faster) To update and reload haproxy without replacing config files.
Also, if you will start the container with new backend server, or use another data-api-plane script to add/delete them. This script will still work as expected as it will run on your existing version of haproxy config.
However it will be much better to configure the entire Haproxy using the data-plane-api.