tooolbox/blog/Traefik Routing Configuration Error: Middleware Chains and Docker Label Conflicts
troubleshooting2025-01-15Β·8Β·229/348

Traefik Routing Configuration Error: Middleware Chains and Docker Label Conflicts

Debugging Traefik routing failures caused by middleware ordering issues, Docker label conflicts, and incorrect entrypoint configurations.

Introduction

Traefik has become my preferred reverse proxy for Docker-based deployments because of its automatic service discovery. Instead of manually configuring upstream servers, you add labels to your Docker containers and Traefik picks them up automatically. It sounds magical, and when it works, it is. But when it does not work, debugging can be frustrating because there is no single configuration file to examine β€” the configuration is distributed across Docker labels, Traefik static config, and dynamic config.

I recently deployed a new service and spent two hours trying to figure out why Traefik was returning 404 errors despite the Docker labels being correctly set. The issue was a subtle interaction between middleware ordering and an entrypoint configuration that I had forgotten about.

Environment

  • Traefik version: 3.0.4
  • Docker version: 24.0.7
  • Deployment: Docker Compose with 6 services
  • Entry points: web (HTTP), websecure (HTTPS)
  • Certificate resolver: Let's Encrypt

Problem

I added a new service to my Docker Compose stack with the following labels:

services:
  new-api:
    image: myregistry/new-api:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.new-api.rule=Host(`api.example.com`)"
      - "traefik.http.routers.new-api.entrypoints=websecure"
      - "traefik.http.routers.new-api.tls.certresolver=letsencrypt"
      - "traefik.http.services.new-api.loadbalancer.server.port=8080"
      - "traefik.http.routers.new-api.middlewares=rate-limit@docker,auth@docker"
    networks:
      - web

When I deployed the service:

docker compose up -d new-api

Traefik showed the following in its dashboard:

routers:
  new-api:
    status: enabled
    rule: Host(`api.example.com`)
    entrypoints: [websecure]
    middlewares: [rate-limit@docker, auth@docker]
    service: new-api
    tls:
      certresolver: letsencrypt

Everything looked correct in the dashboard. But accessing https://api.example.com returned a 404 error. I checked the Traefik logs:

time="2025-01-15T10:30:00Z" level=debug msg="Testing configuration" routerName=new-api@docker
time="2025-01-15T10:30:00Z" level=info msg="Configuration loaded" providerName=docker
time="2025-01-15T10:30:01Z" level=debug msg="Handling connection" entryPointName=websecure routerName=new-api@docker
time="2025-01-15T10:30:01Z" level=debug msg="Evaluating middleware" middlewareName=rate-limit@docker routerName=new-api@docker
time="2025-01-15T10:30:01Z" level=debug msg="Evaluating middleware" middlewareName=auth@docker routerName=new-api@docker
time="2025-01-15T10:30:01Z" level=error msg="middleware not found" middlewareName=auth@docker routerName=new-api@docker

The last line was the key β€” the auth@docker middleware was not found. Traefik was trying to apply a middleware that did not exist.

Analysis

I checked all the middleware definitions in the Docker Compose stack:

services:
  traefik:
    image: traefik:v3.0
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
    labels:
      - "traefik.enable=true"

      # Rate limit middleware
      - "traefik.http.middlewares.rate-limit.ratelimit.average=100"
      - "traefik.http.middlewares.rate-limit.ratelimit.burst=50"
      - "traefik.http.middlewares.rate-limit.ratelimit.period=1s"

      # Note: No auth middleware defined here!

The rate-limit middleware was defined, but the auth middleware was missing from the Traefik container labels. The new-api service referenced auth@docker, but no service had defined a middleware with that name.

In Traefik, middleware is defined as Docker labels on any container in the stack. The middleware name must match exactly, including the @docker provider suffix when referenced from another service. The auth middleware was supposed to be defined on a different container, but it had been removed during a previous refactor.

I also discovered a second issue β€” the entrypoint configuration. The Traefik static config had:

- "--entrypoints.websecure.address=:443"

But the new-api router was only configured for websecure. If someone accessed http://api.example.com (HTTP), there was no router to handle it. I needed either a redirect or a separate HTTP router.

Solution

I fixed both issues by adding the missing middleware definition and configuring proper HTTP-to-HTTPS redirection:

services:
  traefik:
    image: traefik:v3.0
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
    labels:
      - "traefik.enable=true"

      # Rate limit middleware
      - "traefik.http.middlewares.rate-limit.ratelimit.average=100"
      - "traefik.http.middlewares.rate-limit.ratelimit.burst=50"
      - "traefik.http.middlewares.rate-limit.ratelimit.period=1s"

      # Basic auth middleware
      - "traefik.http.middlewares.auth.basicauth.users=admin:$$apr1$$xyz$$hashedpassword"

      # Redirect HTTP to HTTPS
      - "traefik.http.middlewares.redirect-https.redirectscheme.scheme=https"

And updated the new-api service labels:

  new-api:
    image: myregistry/new-api:latest
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.new-api.rule=Host(`api.example.com`)"
      - "traefik.http.routers.new-api.entrypoints=websecure"
      - "traefik.http.routers.new-api.tls.certresolver=letsencrypt"
      - "traefik.http.services.new-api.loadbalancer.server.port=8080"
      - "traefik.http.routers.new-api.middlewares=rate-limit@docker,auth@docker"
      # HTTP router with redirect
      - "traefik.http.routers.new-api-insecure.rule=Host(`api.example.com`)"
      - "traefik.http.routers.new-api-insecure.entrypoints=web"
      - "traefik.http.routers.new-api-insecure.middlewares=redirect-https@docker"
    networks:
      - web

After redeploying:

docker compose up -d

Traefik logs confirmed the middleware was now found:

time="2025-01-15T11:00:00Z" level=debug msg="Evaluating middleware" middlewareName=rate-limit@docker routerName=new-api@docker
time="2025-01-15T11:00:00Z" level=debug msg="Evaluating middleware" middlewareName=auth@docker routerName=new-api@docker
time="2025-01-15T11:00:00Z" level=debug msg="Adding backend" backendName=new-api@docker

I created a validation script to check all middleware references before deployment:

import yaml
import re
import sys

def validate_traefik_config(compose_file):
    with open(compose_file) as f:
        config = yaml.safe_load(f)

    defined_middlewares = set()
    referenced_middlewares = set()

    for service_name, service in config.get('services', {}).items():
        labels = service.get('labels', [])
        for label in labels:
            if isinstance(label, str):
                # Find middleware definitions
                match = re.match(r'traefik\.http\.middlewares\.([^.]+)\.', label)
                if match:
                    defined_middlewares.add(match.group(1))

                # Find middleware references
                match = re.match(r'traefik\.http\.routers\.[^.]+\.middlewares=(.*)', label)
                if match:
                    refs = match.group(1).split(',')
                    for ref in refs:
                        name = ref.strip().split('@')[0]
                        referenced_middlewares.add(name)

    missing = referenced_middlewares - defined_middlewares
    if missing:
        print(f"ERROR: Undefined middlewares referenced: {missing}")
        return False

    print("All middleware references are valid")
    return True

if __name__ == "__main__":
    success = validate_traefik_config(sys.argv[1])
    sys.exit(0 if success else 1)
python validate-traefik.py docker-compose.yml
# All middleware references are valid

Lessons Learned

  1. Middleware is defined by labels β€” In Traefik with Docker provider, middleware is defined as labels on any container. If the container with the middleware definition is removed, the middleware disappears.
  2. Check Traefik logs for "middleware not found" β€” This is the most common error when middleware references are broken. Always check the logs when a router returns unexpected errors.
  3. Define all middlewares on the Traefik container β€” To avoid dependency on other containers, define shared middleware (rate-limit, auth, redirects) on the Traefik container itself.
  4. Always handle HTTP-to-HTTPS redirect β€” If you have an HTTPS router, add a corresponding HTTP router with a redirect middleware. Otherwise, HTTP requests will get no response.
  5. Validate before deploying β€” Write a script to check middleware references in your Docker Compose file before deploying. Catching errors early saves debugging time.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.