Istio Service Mesh Setup: mTLS, Traffic Management, and Observability

Introduction

A service mesh adds a transparent infrastructure layer to your Kubernetes cluster that handles service-to-service communication. Istio is the most feature-rich service mesh available, providing mutual TLS, traffic management, load balancing, and observability without changing your application code.

I decided to add Istio to my trading platform cluster to get mutual TLS between services and fine-grained traffic control for canary deployments. The installation itself was straightforward, but configuring it correctly — especially mTLS policies and traffic routing — required careful attention to detail. This post covers the complete setup process and the issues I encountered.

Environment

• Kubernetes: v1.28.4 (k3s, 3 nodes)
• Istio version: 1.20.2
• Installation profile: default
• Services: 8 microservices, 3 of which are critical (api-server, order-service, payment-service)
• Goal: mTLS everywhere, canary deployments, distributed tracing

Problem

After installing Istio with the default profile and enabling sidecar injection on the trading namespace:

istioctl install --set profile=default -y
kubectl label namespace trading istio-injection=enabled
kubectl rollout restart deployment -n trading

I encountered two issues:

1. Services could not communicate — After the sidecar injection, all inter-service calls started failing with connection timeouts.

2. mTLS was not working — Even after applying a strict mTLS policy, plain text connections were still being accepted.

The first issue was immediately visible — the application was returning 500 errors and the logs showed connection refused errors. The second issue was discovered during a security audit.

I checked the sidecar injection:

kubectl get pods -n trading
# NAME                          READY   STATUS    RESTARTS   AGE
# api-server-5f8b9c6d7-x2k4m   2/2     Running   0          5m
# order-service-7d9e8f3a1-k9j2  2/2     Running   0          5m
# payment-service-4b2c1d3e5-m3n 1/2     Running   0          5m

The payment-service pod was showing 1/2 ready — the sidecar container was failing to start. The init container that configures iptables rules was failing:

kubectl logs payment-service-4b2c1d3e5-m3n -c istio-init
# 2024-10-01T10:00:00Z error: failed to apply iptables rules:
# iptables --wait -t nat -A ISTIO_INBOUND -p tcp -m tcp --dport 15006 -j REDIRECT --to-ports 15006
# iptables: No chain/target/match by that name.

The iptables rules were failing because the kernel modules required by Istio were not loaded on the host.

Analysis

The iptables error pointed to a missing kernel module. Istio's init container uses iptables to redirect traffic through the sidecar proxy, and it requires specific kernel modules:

# Check loaded modules
lsmod | grep -E "nf_nat|nf_conntrack|xt_REDIRECT"
# nf_nat                 45056  0
# nf_conntrack          147456  5 nf_nat,nf_defrag_ipv4

The xt_REDIRECT module was missing. This module is required for the REDIRECT target in iptables. I needed to load it:

sudo modprobe xt_REDIRECT
lsmod | grep xt_REDIRECT
# xt_REDIRECT           16384  0

After loading the module, the sidecar init container completed successfully on the payment-service pod.

But the mTLS issue was separate. I checked the Istio policy:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

This policy should enforce mTLS for all services in the mesh. But when I tested with a plain HTTP request from inside a pod:

kubectl exec -it api-server-5f8b9c6d7-x2k4m -c api-server -- \
    curl -v http://order-service:8080/health
# HTTP/1.1 200 OK
# (response received successfully)

Plain text connections were still working! The mTLS policy was not being enforced.

The issue was that the PeerAuthentication policy in istio-system namespace only applies to sidecar proxies that have the policy loaded. The sidecar proxies on the trading namespace were not receiving the policy because of a namespace mismatch.

Solution

I fixed both issues:

1. Load required kernel modules on all nodes:

#!/bin/bash
# setup-istio-kernel-modules.sh — Run on each node

MODULES=(
    "xt_REDIRECT"
    "xt_mark"
    "xt_conntrack"
    "xt_owner"
    "xt_statistic"
    "nf_nat"
    "nf_conntrack"
    "nf_defrag_ipv4"
    "ip_tables"
    "ip_vs"
    "ip_vs_rr"
    "ip_vs_wrr"
    "ip_vs_sh"
    "nf_conntrack_ipv4"
)

for mod in "${MODULES[@]}"; do
    sudo modprobe "$mod" 2>/dev/null || echo "Warning: Could not load $mod"
done

# Persist modules
cat <

2. Configure mTLS correctly:

# istio-system namespace — default for the mesh
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

---
# trading namespace — explicit enforcement
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: trading
spec:
  mtls:
    mode: STRICT

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: default
  namespace: trading
spec:
  host: "*.trading.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

# VirtualService for weighted routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: api-server
  namespace: trading
spec:
  hosts:
    - api-server
  http:
    - route:
        - destination:
            host: api-server
            subset: stable
          weight: 90
        - destination:
            host: api-server
            subset: canary
          weight: 10

---
# DestinationRule with subsets
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: api-server
  namespace: trading
spec:
  host: api-server
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 100
        http2MaxRequests: 1000
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
  subsets:
    - name: stable
      labels:
        version: v1.2.0
    - name: canary
      labels:
        version: v1.3.0

# Install the add-ons
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml

# Configure Istio for tracing
istioctl install --set profile=default \
    --set meshConfig.enableTracing=true \
    --set meshConfig.defaultConfig.tracing.zipkin.address=jaeger-collector.istio-system:9411 \
    --set meshConfig.defaultConfig.tracing.sampling=100.0

# Verify mTLS is working
istioctl authn tls-check api-server.trading.svc.cluster.local order-service.trading.svc.cluster.local
# Host: order-service.trading.svc.cluster.local:8080
#    Port: 8080/HTTP using Istio identity

# Verify sidecar injection
kubectl get pods -n trading -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.containers[*]}{.name}{" "}{end}{"\n"}{end}'
# api-server-5f8b9c6d7-x2k4m     api-server istio-proxy
# order-service-7d9e8f3a1-k9j2   order-service istio-proxy
# payment-service-4b2c1d3e5-m3n  payment-service istio-proxy