App-of-Apps pattern · Prasanta Kumar Behera

Bootstrap a whole cluster from one Git commit with the App-of-Apps pattern: a root Application that manages child Applications, a sensible repository layout for platform add-ons and apps, AppProjects as guard rails, and how deletion cascades.

One commit to bootstrap a cluster

A new cluster needs many things before apps can run: ingress, cert-manager, monitoring, logging, policies, then the apps. Instead of installing each by hand, create one root Application pointing at a folder of Application manifests:

gitops/
├── bootstrap/
│   └── root.yaml            # applied once by hand
├── apps/                    # the root app points here
│   ├── cert-manager.yaml    # each file is an Application
│   ├── ingress-nginx.yaml
│   ├── monitoring.yaml
│   └── shop.yaml
└── manifests/
    ├── monitoring/          # what the child apps deploy
    └── shop/

Instead of giving the robot 20 separate notes, you give it one note: "Read the notes in this folder and follow them all." Adding a new job is just dropping another note into the folder.

The root Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: https://git.example.com/platform/gitops.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd          # child Application objects live in argocd
  syncPolicy:
    automated: { prune: true, selfHeal: true }

A child, e.g. apps/cert-manager.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: platform
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.16.2        # pin chart versions
    helm:
      valuesObject:
        crds:
          enabled: true
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [ CreateNamespace=true ]

(Chart value names change between chart versions; check the chart's values.yaml for the version you pin.)

AppProjects: guard rails

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shop
  namespace: argocd
spec:
  sourceRepos:
    - https://git.example.com/shop/*
  destinations:
    - server: https://kubernetes.default.svc
      namespace: shop-*
  clusterResourceWhitelist: []          # no cluster-scoped objects
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota               # quotas are set by the platform team

The platform project can allow cluster-scoped kinds (CRDs, ClusterRoles); team projects usually can't. Keep the default project locked down or unused.

Deletion and the finalizer

With resources-finalizer.argocd.argoproj.io, deleting an Application deletes its resources (cascade).
Without it, the resources stay (orphaned but running).
With App-of-Apps and prune, removing a child file from Git deletes that whole app. That's powerful and dangerous; see the scenario in lesson 07 and use Prune=false or Delete=false annotations on critical resources.

Try it: bootstrap a cluster from Git

Create the gitops/ layout above in your repo, with two child apps (e.g. a small nginx app and podinfo from its Helm chart).
Apply only bootstrap/root.yaml; watch the children appear in the UI.
Add a third child file, commit, push; watch it appear without touching the cluster.
Create a shop AppProject that allows only shop-* namespaces; point a child at namespace default and read the error.
Delete a kind cluster, create a new one, install Argo CD, apply the root app again: the whole cluster rebuilds from Git.

Going deeper: repo strategy

Separate repos for platform add-ons and for team apps (different reviewers, different change rates), or one monorepo with CODEOWNERS per folder.
Keep environment differences in overlays or values files (lesson 04), not copies of whole folders.
When the list of apps is generated from data (clusters, folders), switch to ApplicationSets (lesson 05); App-of-Apps suits hand-curated lists.

Recap

App-of-Apps: a root Application manages a folder of child Applications; bootstrap = one kubectl apply.
AppProjects restrict repos, destinations and resource kinds; don't leave everything in default.
The resources finalizer makes deletion cascade; remember that removing a child from Git removes the app.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.