tooolbox/blog/Consul Service Discovery: Health Checks and Service Mesh Integration
architecture2024-12-01Β·9Β·245/348

Consul Service Discovery: Health Checks and Service Mesh Integration

Implementing Consul for service discovery with health checks, DNS resolution, and integration with microservices in a Docker environment.

Introduction

Service discovery is one of those problems that does not seem like a problem until you have more than a handful of services. When I started with two or three microservices, I just hardcoded the hostnames in each service's configuration. But as the system grew to 12 services with multiple instances of each, hardcoded addresses became unmanageable. Services were being added and removed dynamically, ports were changing, and I needed a way for services to find each other without configuration changes.

I chose HashiCorp Consul because it provides both service discovery and health checking out of the box. It also has a DNS interface, which means services can discover each other using standard DNS resolution without any client library changes. This post covers my Consul setup and the issues I resolved during implementation.

Environment

  • Consul version: 1.17.1
  • Deployment: Docker Compose with Consul server and client agents
  • Nodes: 3 server agents, 12 client agents (one per service)
  • Services: Go, Python, Node.js microservices
  • Health checks: HTTP endpoint checks, TCP port checks, TTL checks

Problem

After deploying Consul and registering services, the health checks were failing for services running inside Docker containers. The Consul UI showed most services as "critical" even though they were running fine:

consul catalog services
# api-gateway (healthy)
# user-service (critical)
# order-service (critical)
# payment-service (critical)
# notification-service (critical)

Only the API gateway was healthy. The others were marked critical because their health checks were failing. I checked the health check details:

consul health -service=user-service
# Node          Address        Status  Notes
# docker-node1  10.0.0.11:8080 critical  HTTP GET http://10.0.0.11:8500/health: 500 Internal Server Error

Wait β€” the health check URL was wrong. It was checking http://10.0.0.11:8500/health instead of the actual service port. The health check was configured to use Consul's own port (8500) instead of the service port.

Additionally, I discovered that Consul DNS resolution was not working from inside Docker containers. The containers were using Docker's default DNS (127.0.0.11), which does not know about Consul.

Analysis

I examined the service registration that was being used:

import requests

def register_service():
    registration = {
        "Name": "user-service",
        "ID": "user-service-1",
        "Address": "10.0.0.11",
        "Port": 8080,
        "Check": {
            "HTTP": "http://10.0.0.11:8500/health",  # Wrong port!
            "Interval": "10s",
            "Timeout": "3s"
        }
    }

    response = requests.put(
        "http://localhost:8500/v1/agent/service/register",
        json=registration
    )
    return response.status_code

The health check URL was hardcoded to port 8500 (Consul's port) instead of the service's actual port (8080). This was a simple mistake, but it was being made by every service because they were all using the same registration template.

The second issue was DNS resolution. Consul provides DNS on port 8600, but Docker containers use Docker's embedded DNS server at 127.0.0.11. To use Consul DNS from containers, I needed to configure Docker to forward DNS queries to Consul.

Solution

I addressed both issues with the following changes:

1. Fix health check registration:

I created a shared Python module for service registration:

# consul_registration.py
import os
import socket
import requests
from functools import lru_cache

CONSUL_HOST = os.environ.get("CONSUL_HOST", "consul")
CONSUL_PORT = int(os.environ.get("CONSUL_PORT", "8500"))
SERVICE_PORT = int(os.environ.get("SERVICE_PORT", "8080"))
SERVICE_NAME = os.environ.get("SERVICE_NAME", "unknown-service")

@lru_cache(maxsize=1)
def get_container_ip():
    hostname = socket.gethostname()
    ip = socket.gethostbyname(hostname)
    return ip

def register_service(service_id=None, tags=None):
    container_ip = get_container_ip()
    service_id = service_id or f"{SERVICE_NAME}-{container_ip}"

    registration = {
        "Name": SERVICE_NAME,
        "ID": service_id,
        "Address": container_ip,
        "Port": SERVICE_PORT,
        "Tags": tags or [],
        "Check": {
            "HTTP": f"http://{container_ip}:{SERVICE_PORT}/health",
            "Interval": "10s",
            "Timeout": "3s",
            "DeregisterCriticalServiceAfter": "30s"
        },
        "Meta": {
            "version": os.environ.get("APP_VERSION", "unknown"),
            "environment": os.environ.get("ENVIRONMENT", "development")
        }
    }

    response = requests.put(
        f"http://{CONSUL_HOST}:{CONSUL_PORT}/v1/agent/service/register",
        json=registration,
        timeout=5
    )

    if response.status_code == 200:
        print(f"Registered {SERVICE_NAME} at {container_ip}:{SERVICE_PORT}")
    else:
        print(f"Failed to register: {response.status_code} {response.text}")

    return response.status_code

def deregister_service(service_id=None):
    container_ip = get_container_ip()
    service_id = service_id or f"{SERVICE_NAME}-{container_ip}"

    response = requests.put(
        f"http://{CONSUL_HOST}:{CONSUL_PORT}/v1/agent/service/deregister/{service_id}",
        timeout=5
    )
    return response.status_code

Each service now registered with the correct health check URL using its own port.

2. Configure Consul DNS forwarding in Docker:

I updated the Docker daemon configuration on each host:

{
  "dns": ["172.17.0.1", "8.8.8.8"],
  "dns-search": ["consul.service.consul"]
}

And created a Consul DNS configuration that forwards to upstream DNS:

# /etc/consul.d/dns-config.hcl
dns_config {
  allow_stale = true
  max_stale = "10s"
  node_ttl = "10s"
  service_ttl = "10s"
  enable_redirects = true
}

For services that need to discover other services, I provided a helper function:

# service_discovery.py
import socket
import dns.resolver

CONSUL_DNS = os.environ.get("CONSUL_DNS", "127.0.0.1:8600")

def discover_service(service_name):
    """Resolve a service name to a list of (host, port) tuples."""
    try:
        answers = dns.resolver.resolve(f"{service_name}.service.consul", "A")
        results = []
        for rdata in answers:
            # Get the port from SRV record
            srv_answers = dns.resolver.resolve(f"{service_name}.service.consul", "SRV")
            port = srv_answers[0].port if srv_answers else 8080
            results.append((rdata.address, port))
        return results
    except dns.resolver.NoAnswer:
        return []
    except dns.resolver.NXDOMAIN:
        return []

def get_healthy_service(service_name):
    """Get a single healthy service instance using Consul health API."""
    import requests
    response = requests.get(
        f"http://localhost:8500/v1/health/service/{service_name}?passing=true",
        timeout=5
    )
    services = response.json()
    if services:
        service = services[0]["Service"]
        return (service["Address"], service["Port"])
    return None

I also created a health endpoint for each service that checks downstream dependencies:

# health_check.py
from flask import Flask, jsonify
import requests

app = Flask(__name__)

@app.route("/health")
def health():
    checks = {
        "status": "healthy",
        "checks": {}
    }

    # Check database connectivity
    try:
        # Database health check logic
        checks["checks"]["database"] = "healthy"
    except Exception as e:
        checks["checks"]["database"] = f"unhealthy: {str(e)}"
        checks["status"] = "unhealthy"

    # Check Consul connectivity
    try:
        response = requests.get("http://consul:8500/v1/status/leader", timeout=2)
        if response.status_code == 200:
            checks["checks"]["consul"] = "healthy"
        else:
            checks["checks"]["consul"] = "unhealthy"
            checks["status"] = "unhealthy"
    except Exception:
        checks["checks"]["consul"] = "unhealthy"
        checks["status"] = "unhealthy"

    status_code = 200 if checks["status"] == "healthy" else 503
    return jsonify(checks), status_code

After deploying the fixes, all services showed as healthy in Consul:

consul catalog services
# api-gateway (healthy)
# user-service (healthy)
# order-service (healthy)
# payment-service (healthy)
# notification-service (healthy)

consul members
# Node          Address        Status  Type    Build  Protocol  DC   Segment
# server1       10.0.0.1:8301  alive   server  1.17.1  2        dc1  
# server2       10.0.0.2:8301  alive   server  1.17.1  2        dc1  
# server3       10.0.0.3:8301  alive   server  1.17.1  2        dc1

Lessons Learned

  1. Health checks must hit the right port β€” The most common Consul registration mistake is using the wrong port in health check URLs. Use environment variables for the service port.
  2. Docker DNS needs configuration β€” By default, Docker does not forward DNS queries to Consul. Configure the Docker daemon or use Consul's DNS interface on a well-known address.
  3. Deregister on shutdown β€” Always deregister services when they shut down. Use DeregisterCriticalServiceAfter as a safety net for ungraceful shutdowns.
  4. Health endpoints should check dependencies β€” A health endpoint that only returns 200 is not useful. Check database, cache, and external service connectivity.
  5. Use Consul KV for configuration β€” Consul's key-value store is useful for runtime configuration that needs to be shared across services.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.