tooolbox/blog/Linkerd Migration Problems: Zero-Downtime Migration from Istio
migration2024-09-15Β·9Β·270/348

Linkerd Migration Problems: Zero-Downtime Migration from Istio

Migrating from Istio to Linkerd service mesh with zero downtime, handling configuration differences, and resolving compatibility issues.

Introduction

After running Istio for about six months on my trading platform cluster, I decided to migrate to Linkerd. The reasons were practical: Istio's resource consumption was too high for my small cluster (3 nodes, 24GB total RAM), and the complexity of Istio's configuration was overkill for what I actually needed β€” mutual TLS and basic traffic management.

Linkerd is lighter, simpler, and purpose-built for Kubernetes. But migrating between service meshes is not a simple swap. Both meshes inject sidecar proxies that intercept all network traffic, and having two proxies in the same pod causes conflicts. The migration required careful planning to avoid downtime.

Environment

  • Source mesh: Istio 1.20.2
  • Target mesh: Linkerd 2.14.6
  • Kubernetes: v1.28.4 (k3s)
  • Services: 8 microservices in the trading namespace
  • Constraint: Zero downtime during migration

Problem

The migration plan was:

  1. Uninstall Istio from the namespace (removing sidecars)
  2. Install Linkerd
  3. Enable Linkerd injection on the namespace
  4. Restart pods to inject Linkerd sidecars

The first problem appeared when I tried to remove Istio from the namespace. I removed the istio-injection=enabled label and restarted the pods:

kubectl label namespace trading istio-injection-
kubectl rollout restart deployment -n trading

The pods restarted without Istio sidecars, which was expected. But several services immediately started failing because they depended on Istio-specific features:

  1. VirtualService routing rules were no longer applied
  2. Istio mTLS policies caused connection rejections
  3. Prometheus metrics from Istio telemetry stopped flowing

The second problem appeared when I tried to install Linkerd in the same namespace:

linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
kubectl label namespace trading linkerd.io/injection=enabled
kubectl rollout restart deployment -n trading

Some pods failed to start because the Linkerd proxy-injector could not pull the proxy image:

Warning  Failed     3m   kubelet  Failed to pull image "cr.l5d.io/linkerd/proxy:2.14.6": rpc error: code = Unknown desc = failed to pull and unpack image

The Linkerd image registry (cr.l5d.io) was not accessible from my Kubernetes nodes. This turned out to be a DNS resolution issue β€” the nodes were using a DNS resolver that could not resolve the Linkerd container registry domain.

Analysis

I diagnosed the DNS issue by testing resolution from the affected nodes:

# On a Kubernetes node
nslookup cr.l5d.io
# server can't find cr.l5d.io: NXDOMAIN

# Test with a public DNS server
nslookup cr.l5d.io 8.8.8.8
# Name: cr.l5d.io
# Address: 104.18.0.0 (resolved)

The local DNS resolver could not resolve cr.l5d.io, but Google's DNS could. The local resolver was a Pi-hole instance that was blocking the domain because it was classified as a "container registry" and the Pi-hole list was aggressive about blocking telemetry-related domains.

I also discovered that Linkerd requires specific CRDs to be installed before the control plane:

kubectl get crd | grep linkerd
# (empty β€” CRDs were not installed)

The CRD installation command failed silently because of a network timeout when trying to pull the CRD manifests from the Linkerd GitHub releases.

Solution

I resolved the issues in this order:

1. Fix DNS resolution for the Linkerd registry:

# Add public DNS as fallback in /etc/resolv.conf
# On each node, edit /etc/resolv.conf
nameserver 127.0.0.1
nameserver 8.8.8.8

# Or configure the DNS in k3s
# /etc/rancher/k3s/config.yaml
dns:
  - 8.8.8.8
  - 1.1.1.1

2. Install Linkerd CRDs manually:

Since the automatic install was timing out, I downloaded the CRD manifests locally and applied them:

# Download CRD manifests
curl -fsL https://github.com/linkerd/linkerd2/releases/download/stable-2.14.6/linkerd-crds.yaml -o linkerd-crds.yaml

# Apply CRDs
kubectl apply -f linkerd-crds.yaml

# Verify CRDs are installed
kubectl get crd | grep linkerd
# certificates.linkerd.io
# destinations.linkerd.io
# healthchecks.linkerd.io
# meshtlsauthentications.linkerd.io
# profiles.linkerd.io
# serviceprofiles.linkerd.io

3. Install Linkerd control plane:

# Download control plane manifest
curl -fsL https://github.com/linkerd/linkerd2/releases/download/stable-2.14.6/linkerd.yaml -o linkerd.yaml

# Apply control plane
kubectl apply -f linkerd.yaml

# Verify control plane health
linkerd check

4. Configure namespace injection and restart pods:

# Enable injection on the trading namespace
kubectl label namespace trading linkerd.io/injection=enabled

# Restart pods one at a time to avoid downtime
for deploy in api-server order-service payment-service worker dashboard notification nginx; do
    kubectl rollout restart deployment/$deploy -n trading
    kubectl rollout status deployment/$deploy -n trading --timeout=120s
    echo "Completed rollout for $deploy"
done

5. Convert Istio resources to Linkerd equivalents:

Istio VirtualService resources do not work with Linkerd. I created Linkerd ServiceProfile resources instead:

# Istio VirtualService (removed)
# apiVersion: networking.istio.io/v1beta1
# kind: VirtualService
# ...

# Linkerd ServiceProfile (new)
apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: api-server.trading.svc.cluster.local
  namespace: trading
spec:
  routes:
    - name: GET /api/v1/orders
      condition:
        method: GET
        pathRegex: /api/v1/orders
    - name: POST /api/v1/orders
      condition:
        method: POST
        pathRegex: /api/v1/orders

6. Update Prometheus to scrape Linkerd metrics:

# prometheus-config.yaml
scrape_configs:
  - job_name: linkerd-controller
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - linkerd
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
        action: keep
        regex: .+
      - source_labels:
          - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: admin-http

  - job_name: linkerd-proxy
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_container_name
          - __meta_kubernetes_pod_container_port_name
        separator: ;
        regex: linkerd-proxy;linkerd-admin
        action: keep

After completing the migration:

# Verify Linkerd is working
linkerd check --namespace trading
# ...
# Status check results are OK

# Verify mTLS is active
linkerd stat deploy -n trading
# NAME               MESHED   SUCCESS   RPS    LATENCY_P50   LATENCY_P99   STATUS
# api-server         1/1      100.0%   45.2   12ms          89ms          -
# order-service      1/1      100.0%   23.1   8ms           45ms          -
# payment-service    1/1      99.8%    12.4   15ms          120ms         -

All services were meshed (1/1), had high success rates, and Linkerd was providing mTLS and observability.

Lessons Learned

  1. Plan the migration carefully β€” Removing one mesh and adding another requires coordinated pod restarts. Do them one service at a time.
  2. Check DNS and network access β€” Container registries for service mesh images may be blocked by corporate DNS or firewalls. Verify access before starting the migration.
  3. Convert mesh-specific resources β€” Istio VirtualService and DestinationRule do not work with Linkerd. Convert them to ServiceProfile and TrafficPolicy resources.
  4. Update monitoring configuration β€” Service mesh metrics come from different ports and labels depending on the mesh. Update your Prometheus scrape configs accordingly.
  5. Run linkerd check β€” Linkerd has a built-in health check tool that validates the entire installation. Run it after every change to ensure the mesh is healthy.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.