The multi-provider chicken-and-egg

The problem

You want one terraform apply to create an EKS cluster and install things into it (the AWS Load Balancer Controller, Karpenter, namespaces). But the kubernetes and helm providers need the cluster's endpoint, CA and credentials at configuration time, before the cluster exists.

Symptoms you'll see: plan errors about connecting to localhost, providers using the wrong cluster, or applies that fail after about 15 minutes with Unauthorized.

Fix 1: configure providers from outputs, with exec auth

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}

• exec asks aws eks get-token for a token whenever the provider needs one, so long applies don't hit the ~15-minute expiry of a pre-fetched token.
• The machine running Terraform needs the AWS CLI and credentials that map to cluster access (access entries, lesson 05).
• (Newer major versions of the Helm provider changed the kubernetes block syntax; check the docs for the version you pin.)

This works in one stack when the cluster already exists. For the very first apply, or cluster replacement, it's still fragile.

Fix 2 (recommended): separate layers

live/prod/cluster/    → VPC refs, EKS, node groups, IAM, EKS add-ons      (aws provider only)
live/prod/platform/   → Helm releases, namespaces, CRDs, Karpenter config (kubernetes + helm providers)

The platform layer reads the cluster layer's outputs:

data "terraform_remote_state" "cluster" {
  backend = "s3"
  config = {
    bucket = "acme-tfstate-prod"
    key    = "eks/prod/cluster/terraform.tfstate"
    region = "eu-west-1"
  }
}

locals {
  cluster_name     = data.terraform_remote_state.cluster.outputs.cluster_name
  cluster_endpoint = data.terraform_remote_state.cluster.outputs.cluster_endpoint
  cluster_ca       = data.terraform_remote_state.cluster.outputs.cluster_certificate_authority_data
}

Apply the cluster layer, then the platform layer. Each has a smaller state, a clear owner and a clean plan.

Fix 3: hand in-cluster things to GitOps

Many teams let Terraform stop at "cluster + IAM + bootstrap Argo CD", and let GitOps install everything else (controllers, policies, apps) from Git (see GitOps with Argo CD). Terraform then never needs long-lived access inside the cluster.

What not to rely on

• -target to bootstrap in two steps: fine in an emergency, but a sign the structure needs splitting.
• depends_on between providers and modules: providers can't truly depend on resources; it hides the problem rather than solving it.

Recap

• Providers need the cluster before they can be configured: that's the chicken-and-egg.
• Use exec authentication (aws eks get-token) to avoid token expiry during long applies.
• Best structure: a cluster layer and a platform layer linked by remote state, or hand in-cluster work to GitOps.
• Treat -target and provider depends_on as smells, not solutions.