tooolbox/blog/ArgoCD GitOps Deployment Error: Sync Failures and Resource Drift
devops2024-11-01Β·9Β·253/348

ArgoCD GitOps Deployment Error: Sync Failures and Resource Drift

Debugging ArgoCD sync failures caused by resource drift, permission issues, and incorrect application definitions in GitOps workflows.

Introduction

GitOps with ArgoCD is an elegant concept: your Git repository is the single source of truth for your Kubernetes deployments, and ArgoCD ensures that the cluster state matches what is defined in Git. Push a commit, and the deployment updates automatically. It sounds perfect, but in practice, there are many ways the sync can fail or produce unexpected results.

I recently set up ArgoCD for a Kubernetes cluster running my trading platform microservices. The initial deployment worked, but subsequent updates were failing with cryptic sync errors. The ArgoCD UI showed the applications as "OutOfSync" but did not clearly explain what was different. After extensive debugging, I found three separate issues contributing to the sync failures.

Environment

  • ArgoCD version: 2.9.3
  • Kubernetes: v1.28.4 (k3s on 3 nodes)
  • Git repository: Private GitHub repo with Kustomize overlays
  • Applications: 8 microservices with separate ArgoCD applications
  • Sync method: Automatic sync with self-heal enabled

Problem

After deploying all applications, I noticed that two of them were stuck in "OutOfSync" status:

argocd app list
# NAME           STATUS     HEALTH   SYNCPOLICY  CONDITIONS
# api-gateway    Synced     Healthy  Auto-Create  
# user-service   OutOfSync  Degraded Auto-Create  ComparisonError
# order-service  OutOfSync  Healthy  Auto-Create  
# payment-svc    Synced     Healthy  Auto-Create

The user-service was both OutOfSync and Degraded, while order-service was OutOfSync but Healthy. I checked the details:

argocd app get user-service
# ...
# Conditions:
#   - Type: ComparisonError
#     Message: 'rpc error: code = Unknown desc = unknown error:
#              unable to recognize "": no matches for kind "Deployment"
#              in version "apps/v1"'

The error mentioned a missing Deployment kind in apps/v1. This was strange because the Deployment manifest was clearly in the Git repository with the correct apiVersion.

For order-service, the situation was different:

argocd app get order-service
# ...
# Status: OutOfSync
# Diff:
#   - name: IMAGE_TAG
#     value: v1.2.3 (live) vs v1.3.0 (desired)

The application was OutOfSync because the live deployment was running image tag v1.2.3 while Git specified v1.3.0. But automatic sync was not updating it β€” the sync policy was set to Auto-Create which only creates new resources, it does not update existing ones.

Analysis

Issue 1: Missing CRD for user-service

The user-service deployment used a custom resource definition (CRD) that was not installed in the cluster. The CRD defined a ServiceMonitor resource for Prometheus metrics:

# This CRD was not installed
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: user-service

ArgoCD was trying to apply the entire Kustomize output, including the ServiceMonitor, but the CRD did not exist in the cluster. The "unable to recognize" error was ArgoCD saying it could not find the ServiceMonitor resource type.

The root cause was that the Prometheus Operator CRDs were installed separately, and the user-service Kustomize overlay included a ServiceMonitor that required the CRD to be present first.

Issue 2: Sync policy mismatch for order-service

The order-service had syncPolicy.auto.create but not auto.prune or auto.selfHeal. The create policy only applies to resources that do not exist in the cluster. For existing resources, ArgoCD detects drift but does not automatically fix it unless selfHeal is enabled.

Issue 3: Resource ordering

ArgoCD applies resources in a specific order (Namespaces, Services, Deployments, etc.), but Kustomize does not guarantee that CRDs are applied before resources that depend on them.

Solution

I fixed all three issues:

1. Ensure CRDs are installed before dependent resources:

I restructured the Git repository to separate CRDs from application manifests:

gitops/
β”œβ”€β”€ base/
β”‚   β”œβ”€β”€ crds/
β”‚   β”‚   β”œβ”€β”€ service-monitor-crd.yaml
β”‚   β”‚   └── prometheus-rules-crd.yaml
β”‚   └── apps/
β”‚       β”œβ”€β”€ api-gateway/
β”‚       β”œβ”€β”€ user-service/
β”‚       └── order-service/
β”œβ”€β”€ clusters/
β”‚   β”œβ”€β”€ production/
β”‚   └── staging/
└── argocd/
    β”œβ”€β”€ root-application.yaml
    └── app-of-apps.yaml

I created a root Application that deploys CRDs first:

# argocd/root-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-resources
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops.git
    targetRevision: main
    path: base/crds
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Then each application depends on the root:

# argocd/app-of-apps.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: user-service
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/gitops.git
    targetRevision: main
    path: base/apps/user-service
  destination:
    server: https://kubernetes.default.svc
    namespace: user-service
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

2. Enable full auto-sync with self-heal:

syncPolicy:
  automated:
    prune: true       # Remove resources not in Git
    selfHeal: true    # Fix drift automatically
    allowEmpty: false # Prevent deleting all resources
  syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
    - ApplyOutOfSyncOnly=true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m

3. Handle application hooks for initialization:

For resources that need specific ordering, I used ArgoCD hooks:

# Pre-sync hook to install CRDs
apiVersion: batch/v1
kind: Job
metadata:
  name: install-crds
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
        - name: crd-installer
          image: bitnami/kubectl:1.28
          command:
            - /bin/sh
            - -c
            - |
              kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml
              kubectl wait --for condition=Established crd/servicemonitors.monitoring.coreos.com --timeout=60s
      restartPolicy: Never
  backoffLimit: 3

After applying the fixes:

argocd app sync cluster-resources
# Application spec is synced

argocd app sync user-service
# STATUS: Synced
# HEALTH: Healthy

argocd app sync order-service
# STATUS: Synced
# HEALTH: Healthy

All applications were now syncing correctly. I verified the sync was working with a test commit:

# Update image tag in Git
git commit -am "chore: bump user-service to v1.4.0"
git push origin main

# ArgoCD auto-syncs within 3 minutes (polling interval)
argocd app get user-service
# STATUS: Synced
# IMAGE: myregistry/user-service:v1.4.0

Lessons Learned

  1. CRDs must be installed before dependent resources β€” ArgoCD cannot apply custom resources if the CRD is not in the cluster. Use PreSync hooks or a root application for CRD management.
  2. Enable selfHeal for GitOps β€” Without selfHeal: true, ArgoCD detects drift but does not fix it. For true GitOps, you need automatic remediation.
  3. Use sync waves for ordering β€” ArgoCD annotations like argocd.argoproj.io/sync-wave control the order of resource deployment.
  4. Server-Side Apply prevents conflicts β€” Enable ServerSideApply to avoid conflicts when multiple controllers modify the same resources.
  5. Monitor sync status β€” Set up ArgoCD notifications to alert when applications go OutOfSync or fail to sync.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.