Agents on Kubernetes · Prasanta Kumar Behera

Run every build in its own short-lived Kubernetes pod: how the Kubernetes plugin launches agents, pod templates with several tool containers, resource requests, caches that survive between builds, isolation per team, and troubleshooting agents that never come online.

How it works

With the Kubernetes plugin, Jenkins creates a pod per build (or per stage), in a namespace you choose. The pod contains:

the jnlp container (the agent, added automatically), which connects back to the controller,
your tool containers (Maven, Node, BuildKit, Terraform...), which run the steps,
a shared workspace volume.

When the build ends, the pod is deleted. Capacity follows the cluster's autoscaler, not a fleet of hand-maintained agent VMs.

Instead of keeping a permanent workshop for every trade, you rent a clean, fully equipped pop-up workshop for each job. When the job's done, it's packed away, and nobody inherits someone else's sawdust.

A pod template in the Jenkinsfile

pipeline {
  agent {
    kubernetes {
      defaultContainer 'python'
      yaml '''
apiVersion: v1
kind: Pod
spec:
  serviceAccountName: jenkins-agent
  containers:
    - name: python
      image: python:3.12-slim
      command: ["sleep"]
      args: ["infinity"]
      resources:
        requests: { cpu: 500m, memory: 1Gi }
        limits:   { memory: 2Gi }
    - name: terraform
      image: hashicorp/terraform:1.12
      command: ["sleep"]
      args: ["infinity"]
'''
    }
  }
  stages {
    stage('Test') { steps { sh 'pip install -r requirements.txt && pytest' } }
    stage('Plan') { steps { container('terraform') { sh 'terraform init -input=false && terraform plan' } } }
  }
}

Tool containers need a long-running command (sleep infinity) so steps can execute in them. Always set requests and limits: the scheduler needs them, and a runaway build shouldn't starve its neighbours.

Central pod templates

Defining the same pod in 40 Jenkinsfiles is copy-paste again. Define pod templates centrally (in JCasC or the cloud configuration) or ship them from the shared library (libraryResource, lesson 05), and reference them with inheritFrom.

Isolation and permissions

Run agents in their own namespace (for example jenkins-agents), with a ResourceQuota.
Give agent pods a ServiceAccount with no cluster permissions by default; deployment rights only for specific, protected pipelines (or none at all with GitOps).
Separate namespaces or node pools for untrusted builds (fork pull requests).
Apply the same Pod Security level you use for applications; rootless image builds (lesson 03) avoid privileged pods.

Keeping ephemeral builds fast

Slowness	Fix
Image pulls for tool containers	Small, pinned images in a nearby registry; pre-pull on nodes or use image caching
Dependencies downloaded every build	A package proxy/cache (Nexus, Artifactory, a pip/npm mirror) in the cluster
Container build layers rebuilt	BuildKit remote cache (`--export-cache`/`--import-cache` to the registry)
Pods waiting for nodes	Cluster autoscaler / Karpenter with a warm buffer during working hours

A shared PVC cache mounted into every pod works for some tools, but concurrent writes and poisoning risks make a proxy or remote cache the safer default.

When agents don't come online

Symptom	Check
Pod stuck `Pending`	`kubectl describe pod`: resources, quota, node selectors, image pull
Pod running, build waits "for an executor"	`jnlp` container logs: can it reach the controller URL and agent port/WebSocket?
Pod killed mid-build	OOMKilled (raise memory limit), or evicted by node pressure
"Unknown container" errors	The `container('name')` doesn't match the pod spec

Try it: ephemeral agents

Install Jenkins with the Helm chart (lesson 01) and confirm the Kubernetes cloud is configured.
Run the pipeline above; watch kubectl get pods -w in the agent namespace as the pod appears and disappears.
Remove the memory limit and run a memory-hungry step, then restore the limit and see the OOM kill in kubectl describe.
Move the pod spec into a central template and use inheritFrom from a second repository.

Recap

The Kubernetes plugin runs each build in a fresh pod: jnlp + your tool containers + a shared workspace.
Always set requests/limits; centralise pod templates.
Agents in their own namespace, minimal ServiceAccount, isolated untrusted builds.
Speed comes from caches and proxies, not long-lived agents.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.