Production patterns · Prasanta Kumar Behera

Run Argo CD like production infrastructure: SSO and RBAC for teams, notifications, automated image updates, high availability and scaling, monitoring Argo CD itself, handling secrets, and limiting the blast radius of a bad commit or an unexpected prune.

SSO and RBAC

Configure SSO in argocd-cm (OIDC directly, or through the bundled Dex for SAML/LDAP/GitHub):

# argocd-cm (data)
url: https://argocd.example.com
oidc.config: |
  name: Corporate SSO
  issuer: https://login.example.com/realms/eng
  clientID: argocd
  clientSecret: $oidc.clientSecret        # a key in argocd-secret
  requestedScopes: [ "openid", "profile", "email", "groups" ]
admin.enabled: "false"                    # after SSO works

# argocd-rbac-cm (data)
policy.default: role:readonly
scopes: '[groups]'
policy.csv: |
  p, role:shop-dev, applications, get,  shop/*, allow
  p, role:shop-dev, applications, sync, shop/*, allow
  p, role:shop-dev, logs,         get,  shop/*, allow
  g, shop-devs, role:shop-dev
  g, platform-admins, role:admin

Combine with AppProjects (lesson 03): projects limit what can be deployed where, and RBAC limits who can act on which project's apps. Keep all of this configuration in Git too.

The robot now works in a big school. Everyone signs in with their school card (SSO). Teachers of class 3 can ask the robot to rebuild class 3's models, but not class 5's (RBAC and projects). The robot rings a bell when a build fails (notifications), and there are two robots in case one breaks (HA).

Notifications

The notifications controller sends messages on triggers (on-sync-failed, on-health-degraded, on-deployed…) to Slack, Teams, email, webhooks and more. Services and templates are configured in argocd-notifications-cm, tokens in argocd-notifications-secret, and apps subscribe with an annotation:

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: platform-alerts
    notifications.argoproj.io/subscribe.on-health-degraded.slack: platform-alerts

Automated image updates

Options for bumping tags without a human editing YAML:

CI commits the tag to the environment overlay (simple, explicit).
Renovate/Dependabot open PRs for new image or chart versions.
Argo CD Image Updater watches registries and writes updates back to Git. Its configuration style has changed between major versions, so follow the docs for the version you install.

Whatever you choose, updates should land as Git commits, so history and rollback stay intact.

HA, scaling and monitoring

Install with the HA manifests or the Helm chart with HA settings: multiple API servers and repo-servers, Redis HA.
Application controller sharding spreads clusters across controller replicas as the fleet grows; the repo-server scales horizontally for heavy Helm/Kustomize rendering.
Scrape Argo CD's metrics. Alert on apps OutOfSync or Degraded for too long (argocd_app_info), sync failures, and repo-server/controller errors and latency.
Back up Argo CD's config (argocd admin export), although with everything declared in Git the real backup is the repo.

Secrets

Never commit plain Secrets. Choose one approach per platform (see Kubernetes Security & Hardening, lesson 11):

Approach	In Git	Decrypted by
Sealed Secrets	Encrypted `SealedSecret`	The in-cluster controller
SOPS (age/KMS)	Encrypted values	A plugin/KSOPS at render time
External Secrets Operator	An `ExternalSecret` reference	ESO, from Vault/AWS Secrets Manager/etc.

Blast radius

Scenario: one bad config, every cluster

A platform engineer merges a change to the shared ingress add-on. An ApplicationSet deploys it to all 40 clusters within minutes, and a typo in the config breaks ingress everywhere.

What should have been in place?

Staged rollout: the change goes to dev clusters, then a small canary group of prod clusters, then the rest, via separate environments, labels such as wave: canary, or ApplicationSet progressive syncs.
CI validation of rendered manifests (kustomize/helm render, schema checks, policy tests) before merge.
Health-gated promotion: the next stage only after the canary is healthy for a while.
Fast rollback: git revert is one commit, and it reaches every cluster through the same path.

Scenario: GitOps pruned something it shouldn't have

A refactor moved an app's manifests to a new folder and a new Application. When the old Application's file was removed from the App-of-Apps folder, Argo CD cascade-deleted the old app including its namespace and PVCs, before the new app was healthy.

How do you prevent it?

Mark data-bearing objects (PVCs, namespaces with data, CRDs) with argocd.argoproj.io/sync-options: Prune=false,Delete=false, or keep them in a separate, rarely-changed app.
For moves, remove the finalizer first (or delete the old Application non-cascading) so resources are orphaned, then let the new app adopt them.
Make deletions stand out in review (CODEOWNERS on bootstrap folders, a CI check that flags removed Applications).
Keep backups (Velero, database backups): GitOps restores config, not data.

Try it: production-grade Argo CD

Set up SSO with a local Keycloak or Dex using static users; map a group to role:shop-dev and verify the user can sync only shop/* apps.
Test your RBAC policy offline with argocd admin settings rbac can.
Configure a notification to a webhook receiver (e.g. a small HTTP echo pod) on on-sync-failed, then break an app.
Install Sealed Secrets or External Secrets and deploy a Secret through GitOps without plain values in Git.
Recreate the prune scenario in a lab: a PVC with Prune=false,Delete=false, delete its Application, and confirm the PVC survives.

Going deeper: platform maturity

Treat Argo CD as tier-0: its own SLOs, upgrades tested on staging, and runbooks.
Standardise app onboarding (a template repo or an ApplicationSet with the SCM provider generator), so teams get projects, RBAC and notifications by default.
Measure DORA metrics (deployment frequency, lead time, change failure rate, time to restore) from Git and Argo CD events.

Recap

SSO (OIDC/Dex) + group-based RBAC + AppProjects; disable the local admin.
Notifications on sync failure and degraded health; image updates always land as Git commits.
HA install, sharding and repo-server scaling; monitor Argo CD itself.
Secrets: Sealed Secrets, SOPS or External Secrets, never plain.
Limit blast radius: staged rollouts, CI validation, Prune=false/Delete=false on data, reviewed deletions, backups.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.