troubleshooting2024-10-01Β·8Β·263/348

Helm Chart Configuration Troubleshooting: Template Rendering and Value Overrides

Debugging Helm chart issues caused by template rendering errors, value override precedence, and chart dependency conflicts.

Introduction

Helm is the package manager for Kubernetes, and it has become the de facto standard for deploying applications. But Helm's template engine, while powerful, has enough quirks to keep you guessing. I recently spent several hours debugging a Helm chart that rendered correctly locally but failed when deployed through ArgoCD.

The issue was a combination of template variable scoping, value override precedence, and a subtle interaction between a parent chart and its dependencies. Here is a detailed walkthrough of the problem and solution.

Environment

  • Helm version: 3.14.0
  • Kubernetes: v1.28.4
  • Chart type: Umbrella chart with 4 sub-charts
  • Deployment: ArgoCD with Helm plugin
  • Values files: Base values + environment-specific overlays

Problem

The umbrella chart had a structure like this:

trading-platform/
β”œβ”€β”€ Chart.yaml
β”œβ”€β”€ values.yaml
β”œβ”€β”€ templates/
β”‚   └── configmap.yaml
β”œβ”€β”€ charts/
β”‚   β”œβ”€β”€ api-server/
β”‚   β”œβ”€β”€ worker/
β”‚   β”œβ”€β”€ dashboard/
β”‚   └── nginx/
└── envs/
    β”œβ”€β”€ production.yaml
    └── staging.yaml

When I ran helm template locally, the output was correct:

helm template trading-platform ./trading-platform \
    -f ./trading-platform/envs/production.yaml
# Rendered configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: trading-platform-config
data:
  DATABASE_HOST: "prod-db.example.com"
  REDIS_HOST: "prod-redis.example.com"
  LOG_LEVEL: "info"

But when ArgoCD deployed the same chart, the ConfigMap had incorrect values:

# What ArgoCD rendered
apiVersion: v1
kind: ConfigMap
metadata:
  name: trading-platform-config
data:
  DATABASE_HOST: "localhost"
  REDIS_HOST: "localhost"
  LOG_LEVEL: "debug"

The values were falling back to the defaults in values.yaml instead of using the production overlay. This caused the application to try connecting to localhost instead of the production database.

Analysis

I examined the template and values files:

# values.yaml
global:
  database:
    host: "localhost"
    port: 5432
  redis:
    host: "localhost"
    port: 6379

config:
  logLevel: "debug"
# envs/production.yaml
global:
  database:
    host: "prod-db.example.com"
    port: 5432
  redis:
    host: "prod-redis.example.com"
    port: 6379

config:
  logLevel: "info"
# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-config
data:
  DATABASE_HOST: {{ .Values.global.database.host | quote }}
  REDIS_HOST: {{ .Values.global.redis.host | quote }}
  LOG_LEVEL: {{ .Values.config.logLevel | quote }}

The templates looked correct. The issue was how ArgoCD was passing values to Helm. I checked the ArgoCD Application manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: trading-platform
spec:
  source:
    repoURL: https://github.com/myorg/charts.git
    path: trading-platform
    targetRevision: main
    helm:
      valueFiles:
        - values.yaml

The problem was in the valueFiles array. ArgoCD was only passing values.yaml and not the production overlay. The production values file was in envs/production.yaml, but ArgoCD was not configured to use it.

There was a second issue. The sub-charts had their own values.yaml files with default database and Redis configurations. When the parent chart's values were not properly passed down, the sub-charts used their own defaults.

I verified by rendering the templates manually:

# What ArgoCD was doing (without production overlay)
helm template trading-platform ./trading-platform \
    -f ./trading-platform/values.yaml

# DATABASE_HOST: "localhost" β€” using defaults
# What it should have been doing
helm template trading-platform ./trading-platform \
    -f ./trading-platform/values.yaml \
    -f ./trading-platform/envs/production.yaml

# DATABASE_HOST: "prod-db.example.com" β€” using production overlay

The solution required changes to both the Helm chart and the ArgoCD Application.

Solution

I made three changes to fix the configuration:

1. Update ArgoCD Application to use environment-specific values:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: trading-platform-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/charts.git
    targetRevision: main
    path: trading-platform
    helm:
      valueFiles:
        - values.yaml
        - envs/production.yaml
      values: |
        global:
          environment: production
  destination:
    server: https://kubernetes.default.svc
    namespace: trading-production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

2. Restructure Helm values to use sub-chart overrides:

The parent chart's values.yaml was restructured to properly pass values to sub-charts:

# values.yaml (parent chart)
global:
  environment: development
  database:
    host: "localhost"
    port: 5432
  redis:
    host: "localhost"
    port: 6379

# Sub-chart value overrides
api-server:
  env:
    DATABASE_HOST: "{{ .Values.global.database.host }}"
    REDIS_HOST: "{{ .Values.global.redis.host }}"
  replicaCount: 1

worker:
  env:
    DATABASE_HOST: "{{ .Values.global.database.host }}"
    REDIS_HOST: "{{ .Values.global.redis.host }}"
  replicaCount: 1

nginx:
  replicaCount: 1

3. Use a helper template for value resolution:

I added a helper template to the parent chart's _helpers.tpl:

{{/*
Resolve a value from global config, allowing per-environment override
*/}}
{{- define "trading-platform.resolveValue" -}}
{{- $root := index . 0 -}}
{{- $key := index . 1 -}}
{{- $default := index . 2 -}}
{{- if hasKey $root.Values.global $key -}}
{{- index $root.Values.global $key -}}
{{- else -}}
{{- $default -}}
{{- end -}}
{{- end -}}

Then used it in the templates:

# templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-config
  labels:
    app: {{ .Release.Name }}
    environment: {{ .Values.global.environment | default "development" }}
data:
  DATABASE_HOST: {{ .Values.global.database.host | default "localhost" | quote }}
  DATABASE_PORT: {{ .Values.global.database.port | default 5432 | quote }}
  REDIS_HOST: {{ .Values.global.redis.host | default "localhost" | quote }}
  REDIS_PORT: {{ .Values.global.redis.port | default 6379 | quote }}
  LOG_LEVEL: {{ .Values.config.logLevel | default "info" | quote }}

I verified the fix by rendering with the production overlay:

helm template trading-platform ./trading-platform \
    -f ./trading-platform/values.yaml \
    -f ./trading-platform/envs/production.yaml
# Rendered output
apiVersion: v1
kind: ConfigMap
metadata:
  name: trading-platform-config
data:
  DATABASE_HOST: "prod-db.example.com"
  DATABASE_PORT: "5432"
  REDIS_HOST: "prod-redis.example.com"
  REDIS_PORT: "6379"
  LOG_LEVEL: "info"

After pushing the changes and syncing through ArgoCD:

argocd app sync trading-platform-production
# STATUS: Synced
# HEALTH: Healthy

# Verify the ConfigMap in the cluster
kubectl get configmap trading-platform-config -n trading-production -o yaml
# DATABASE_HOST: prod-db.example.com
# LOG_LEVEL: info

Lessons Learned

  1. ArgoCD valueFiles must be explicitly listed β€” ArgoCD does not automatically discover values files. You must list every file in the valueFiles array.
  2. Value override order matters β€” Later files in valueFiles override earlier ones. Place environment-specific values last.
  3. Sub-charts have their own defaults β€” When using umbrella charts, ensure parent values are properly passed to sub-charts using chart dependency aliases or global values.
  4. Always test with helm template β€” Before deploying, render the templates locally with the same values files ArgoCD will use to verify the output.
  5. Use environment labels in templates β€” Adding environment labels to resources helps identify which configuration was applied during debugging.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.