HAProxy Load Balancing Configuration: Health Checks and Backend Failover

Introduction

HAProxy is one of those tools that does its job so well that you forget it is there — until something goes wrong. I have been using it as the entry point for a set of microservices, and for the most part, it has been invisible. But when I added a new backend server to the pool, it started receiving traffic before the application was fully initialized, causing intermittent 502 errors for users.

The issue was that my health check configuration was too permissive. HAProxy was marking the server as available based on a simple TCP connection check, which succeeded even though the application inside the container was still starting up. This experience led me to redesign the entire health check and failover strategy, which I am documenting here.

Environment

HAProxy version: 2.8.4
OS: Ubuntu 22.04
Backends: 4 Go microservice instances on Docker
Frontend: Single SSL termination point
Load balancing algorithm: Round-robin (default)

Problem

When I deployed a new backend instance, HAProxy immediately started routing traffic to it. The Go application inside the container needed about 15 seconds to fully initialize (loading configs, establishing database connections, warming caches). During those 15 seconds, approximately 5% of requests to that backend returned 502 errors.

The initial HAProxy configuration was:

global
    log /dev/log local0
    maxconn 4096

defaults
    log     global
    mode    http
    option  httplog
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/example.pem
    default_backend app_servers

backend app_servers
    balance roundrobin
    option httpchk GET /health
    server app1 10.0.0.11:8080 check
    server app2 10.0.0.12:8080 check
    server app3 10.0.0.13:8080 check
    server app4 10.0.0.14:8080 check

The check directive tells HAProxy to perform health checks, but the default behavior is a TCP connection check. Even though I had option httpchk GET /health, the check interval and fall/rise thresholds were at defaults, which were not aggressive enough to catch the initialization period.

Analysis

I examined the HAProxy health check behavior more carefully. The default settings are:

inter 2000 — Check every 2 seconds
fall 3 — Mark as down after 3 consecutive failures
rise 2 — Mark as up after 2 consecutive successes

With a 2-second check interval and a 3-fall threshold, HAProxy would detect a failing server after 6 seconds. But during the initialization period, the /health endpoint was actually responding with HTTP 200 — the Go application was returning a health check response even though it was not fully ready. The health check endpoint was not checking actual readiness.

I verified this by checking the health endpoint during startup:

# During container startup
curl -v http://10.0.0.14:8080/health
# HTTP/1.1 200 OK
# Content-Type: application/json
# {"status": "starting"}

# After full initialization
curl -v http://10.0.0.14:8080/health
# HTTP/1.1 200 OK
# Content-Type: application/json
# {"status": "ready"}

The health endpoint was returning 200 in both cases, with only the response body differing. HAProxy's httpchk only checks the HTTP status code by default, so it saw 200 and considered the server healthy.

The fix required changes on both sides — the application needed to return a non-200 status during initialization, and HAProxy needed to be configured to check the response body.

Solution

I made changes in three areas:

1. Application-side readiness check:

The Go application health endpoint was updated to return proper status codes:

package main

import (
    "net/http"
    "sync/atomic"
)

var (
    ready int32
    mu    sync.Mutex
)

func healthHandler(w http.ResponseWriter, r *http.Request) {
    if atomic.LoadInt32(&ready) == 0 {
        w.WriteHeader(http.StatusServiceUnavailable)
        w.Write([]byte(`{"status":"starting"}`))
        return
    }
    w.WriteHeader(http.StatusOK)
    w.Write([]byte(`{"status":"ready"}`))
}

func main() {
    // ... initialization ...
    atomic.StoreInt32(&ready, 1)
    // ... start server ...
}

2. HAProxy configuration with detailed health checks:

global
    log /dev/log local0
    maxconn 4096
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    option  redispatch
    retries 3

frontend http_front
    bind *:80
    bind *:443 ssl crt /etc/ssl/certs/example.pem
    http-request redirect scheme https unless { ssl_fc }
    default_backend app_servers

backend app_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200

    # Aggressive health checks for faster detection
    default-server inter 1s fall 3 rise 2 slowstart 30s

    server app1 10.0.0.11:8080 check
    server app2 10.0.0.12:8080 check
    server app3 10.0.0.13:8080 check
    server app4 10.0.0.14:8080 check

    # Sticky sessions for stateful operations
    cookie SERVERID insert indirect nocache
    server app1 10.0.0.11:8080 check cookie app1
    server app2 10.0.0.12:8080 check cookie app2
    server app3 10.0.0.13:8080 check cookie app3
    server app4 10.0.0.14:8080 check cookie app4

Key changes:

inter 1s — Health checks every 1 second instead of 2
fall 3 — Mark as down after 3 failures (3 seconds)
slowstart 30s — Gradually increase traffic to new servers
http-check expect status 200 — Explicitly check for HTTP 200
option redispatch — Allow retrying on a different server if one fails mid-request
cookie — Sticky sessions using server cookies

3. Connection draining for graceful shutdown:

backend app_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200

    default-server inter 1s fall 3 rise 2 slowstart 30s on-marked-down shutdown-sessions

    server app1 10.0.0.11:8080 check weight 100
    server app2 10.0.0.12:8080 check weight 100
    server app3 10.0.0.13:8080 check weight 100
    server app4 10.0.0.14:8080 check weight 100

The on-marked-down shutdown-sessions directive immediately closes existing connections to a server when it is marked as down, preventing requests from being sent to a failing server during the transition.

I also created a stats page for monitoring:

listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 5s
    stats auth admin:secure_password

This gives me a real-time view of backend health, connection counts, and request rates at http://localhost:8404/stats.

Lessons Learned

Health checks must verify real readiness — A TCP check or HTTP 200 response is not enough. The health endpoint must verify that the application is actually ready to serve traffic.
Use slowstart for new servers — This gradually ramps up traffic to a newly added server, giving it time to warm up caches and establish connections.
Set aggressive check intervals — In production, 1-second check intervals catch problems quickly. The overhead is minimal compared to the benefit of fast failover.
Configure option redispatch — This allows HAProxy to retry a failed request on a different backend server, reducing error rates during transient failures.
Monitor with the stats page — HAProxy's built-in stats page provides real-time visibility into backend health and traffic distribution.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.