Argo CD GitOps skill for Claude Code

GitOps is working when every change to a cluster arrives as a reviewed commit, the controller reconciles it within minutes, drift is either auto-corrected or explained, no plaintext secret exists in Git, and promoting to production is a pull request that changes one image tag or one directory.

Ground rules

• Git is the source of truth. Never kubectl apply/edit resources owned by Argo CD except during an incident, and then commit the same change immediately.
• Separate the config repo (manifests) from application source repos. CI builds images and opens PRs against the config repo; it does not run kubectl.
• Pin everything: chart versions, image tags (or digests), targetRevision for production apps to a branch or tag you control.
• Each team gets an AppProject that restricts source repos, destination namespaces and cluster-scoped resource kinds. Never give apps the default project in shared clusters.
• Enabling prune: true on an existing app or deleting an Application with a resources finalizer deletes live resources. Confirm with the user and review the diff before either.

Workflow

- [ ] 1. Inspect: controller version, existing apps, repo layout, sync status
- [ ] 2. Lay out the config repo (base + overlays or chart + values per env)
- [ ] 3. Define AppProjects and the bootstrap (app-of-apps or ApplicationSet)
- [ ] 4. Write Applications with explicit sync policy and ordering
- [ ] 5. Handle secrets without plaintext in Git
- [ ] 6. Wire promotion (CI -> PR -> merge) per environment
- [ ] 7. Verify sync, health, and drift; document rollback

1. Inspect

argocd version
argocd app list -o wide
argocd app get <app> --refresh            # sync + health + conditions
argocd app diff <app>                     # live vs desired, exit code 1 when different
kubectl get applications,applicationsets -n argocd
flux get all -A                           # if Flux is in use instead

Argo CD 3.x is current. Notable 3.0 defaults: annotation-based resource tracking, update/delete RBAC no longer implicitly covering sub-resources, logs RBAC enforced. Check release notes before upgrading across minor versions.

2. Repository layout

platform-config/
├── bootstrap/root-app.yaml          # app-of-apps entry point (applied once by hand)
├── projects/                        # AppProject per team
├── apps/                            # Application/ApplicationSet definitions
│   ├── checkout.yaml
│   └── cluster-addons.yaml
└── services/checkout/
    ├── base/                        # kustomize base: deployment, service, hpa
    └── overlays/
        ├── dev/kustomization.yaml   # image tag, replicas, env config
        ├── staging/kustomization.yaml
        └── prod/kustomization.yaml

Choice	Use when
Kustomize overlays per env	In-house services; promotion = change tag in overlay
Helm chart + values-<env>.yaml	Third-party software or a shared internal chart
Directory per env, same branch	Default; avoid branch-per-environment (merges drift and conflict)
Separate repo per env	Strict separation required by compliance; costs more automation

3. Bootstrap and projects

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: team-checkout, namespace: argocd }
spec:
  sourceRepos: ["https://github.com/acme/platform-config.git"]
  destinations:
    - { server: https://kubernetes.default.svc, namespace: "checkout-*" }
  clusterResourceWhitelist: []          # no cluster-scoped kinds
  namespaceResourceBlacklist:
    - { group: "", kind: ResourceQuota }

ApplicationSet generating one app per environment directory:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: checkout, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/acme/platform-config.git
        revision: main
        directories:
          - path: services/checkout/overlays/*
  template:
    metadata:
      name: "checkout-{{.path.basename}}"
    spec:
      project: team-checkout
      source:
        repoURL: https://github.com/acme/platform-config.git
        targetRevision: main
        path: "{{.path.path}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "checkout-{{.path.basename}}"
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true, ServerSideApply=true]

Set spec.syncPolicy.preserveResourcesOnDeletion: true on ApplicationSets in production so deleting the set (or a generator glitch) does not cascade-delete workloads.

Pattern	Use when
App-of-apps (root Application pointing at apps/)	Small number of hand-written apps, explicit control
ApplicationSet git directory generator	One app per folder (services, environments)
ApplicationSet cluster generator	Same add-ons on many clusters
Matrix generator (clusters x directories)	Fleet of clusters each running many apps

4. Applications, sync policy and ordering

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-prod
  namespace: argocd
  finalizers: [resources-finalizer.argocd.argoproj.io]
spec:
  project: team-checkout
  source:
    repoURL: https://github.com/acme/platform-config.git
    targetRevision: main
    path: services/checkout/overlays/prod
  destination: { server: https://kubernetes.default.svc, namespace: checkout-prod }
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ServerSideApply=true, PruneLast=true]
    retry: { limit: 5, backoff: { duration: 10s, factor: 2, maxDuration: 3m } }
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: [/spec/replicas]     # HPA owns replicas

Sync policy decisions:

Setting	Default recommendation
automated	On for dev/staging; on for prod only with PR review on the config repo
selfHeal	On; manual hotfixes get reverted, which is the point
prune	On once the app is stable; review argocd app diff first
ServerSideApply=true	On for large CRDs and fields managed by multiple controllers
Replace=true	Avoid; deletes and recreates

Ordering inside one sync: phases PreSync -> Sync -> PostSync (SyncFail on failure), and within a phase ascending argocd.argoproj.io/sync-wave (default 0, negatives first). Argo waits for each wave to become healthy before the next.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"           # e.g. namespace, CRDs, secrets first
---
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded

Migrations as PreSync hooks must be backward-compatible with the running version (see schema-migrations); a failed hook stops the sync, which is the desired safety.

5. Secrets

Option	Choose when
External Secrets Operator + cloud secret manager	Default for cloud clusters; Git holds only ExternalSecret references
Sealed Secrets	No external secret store; encrypted blobs per cluster in Git
SOPS (age/KMS) via Flux native support or an Argo CD plugin	Already using SOPS; Flux handles it natively

Never use Helm values.yaml with plaintext secrets, and never rely on “private repo” as protection. Rotate any secret that was ever committed.

6. Promotion

Default flow: CI builds image:sha-abc123, then opens a PR that changes the tag in overlays/dev; merge auto-syncs dev. Promotion to staging and prod is a PR copying the same tag to the next overlay, gated by checks and required reviewers.

cd services/checkout/overlays/staging
kustomize edit set image ghcr.io/acme/checkout=ghcr.io/acme/checkout:sha-abc123
git commit -am "checkout: promote sha-abc123 to staging" && gh pr create --fill

For progressive delivery inside one environment, pair with Argo Rollouts (see blue-green-deploys). For automated tag bumps, Argo CD Image Updater or Flux image automation can write commits; keep production promotion a human-reviewed PR.

Detailed promotion and multi-cluster patterns: read references/promotion-patterns.md when designing more than two environments, multiple clusters, or Flux-based promotion.

7. Verify and roll back

argocd app wait checkout-staging --sync --health --timeout 300
argocd app history checkout-prod

Rollback = git revert the promotion commit. argocd app rollback works only with auto-sync disabled and is reverted by the next sync, so use it only as an incident stopgap and follow with a revert commit.

Debugging

Symptom	Likely cause	Action
OutOfSync right after sync	Controller/webhook mutates fields (defaults, HPA replicas, injected sidecars)	argocd app diff; add ignoreDifferences with jsonPointers or managedFieldsManagers
Sync stuck “Progressing”	Health check waits on a resource that never becomes ready	argocd app get resource tree; check pod events; custom Lua health for CRDs
ComparisonError / manifest generation failed	Bad kustomize/helm render, missing repo creds	Render locally: kustomize build / helm template; check repo-server logs
App deleted resources unexpectedly	Prune with a path or generator change	Restore via revert; add Prune=false annotation on critical resources
Changes take minutes to appear	Polling interval (default ~3 min)	Configure a Git webhook to the Argo CD server
“resource not permitted in project”	AppProject whitelist	Extend the project deliberately, not by switching to default

Gotchas

• ignoreDifferences hides drift from the diff, but with RespectIgnoreDifferences=true absent, a sync still overwrites those fields. Add the sync option when another controller owns the field.
• Helm hooks are translated to Argo hooks; helm.sh/hook: pre-install runs on every sync as PreSync, not only on install.
• CRDs and their CRs in the same sync fail on first apply unless CRDs are in an earlier wave or SkipDryRunOnMissingResource=true is set on the CRs.
• Automated sync will not re-sync the same commit after a failure until retry or a new commit; read operationState before assuming it is applied.
• Argo CD itself should be managed by Argo CD (self-managed app) only after the manual bootstrap is reproducible from the repo.

Output

**Apps:** checkout-{dev,staging,prod} via ApplicationSet (git directories)
**Project:** team-checkout, namespaces checkout-*, no cluster-scoped kinds
**Sync:** automated prune+selfHeal; HPA replicas ignored with RespectIgnoreDifferences
**Secrets:** ExternalSecret -> AWS Secrets Manager; none in Git
**Promotion:** CI PR to dev overlay; manual PR dev->staging->prod with required review
**Verified:** argocd app wait healthy in all envs; diff clean; rollback = git revert