Jenkins in the real world · Prasanta Kumar Behera

The Jenkins you'll inherit: declarative Jenkinsfiles, agents on Kubernetes, shared libraries, credentials handling, configuration as code, keeping plugins and the controller healthy, making CI resilient, and migrating gradually to newer systems.

Why Jenkins still matters

Jenkins has run CI for many organisations for well over a decade. You'll meet it in almost every established company: hundreds of jobs, many plugins, and a controller nobody wants to touch. Knowing how to run it well and how to migrate gradually is a practical skill.

Jenkins is the old, reliable school bus. It has been modified by many drivers over the years (plugins), and it still takes everyone to school. You learn how it works, keep it serviced, write down how to rebuild it if it breaks, and plan a calm move to newer buses, without leaving any kids at the bus stop.

A modern declarative Jenkinsfile

@Library('platform-lib@v2') _

pipeline {
  agent {
    kubernetes {
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
    - name: python
      image: registry.example.com/ci/python:3.12
      command: ["sleep"]
      args: ["infinity"]
'''
    }
  }
  options {
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '30'))
    disableConcurrentBuilds()
  }
  stages {
    stage('Test') {
      steps {
        container('python') {
          sh 'pip install -r requirements.txt && pytest -q'
        }
      }
    }
    stage('Publish') {
      when { branch 'main' }
      steps {
        withCredentials([string(credentialsId: 'registry-token', variable: 'TOKEN')]) {
          sh 'echo "$TOKEN" | ./scripts/publish.sh --token-stdin'
        }
      }
    }
  }
  post {
    failure { echo 'Notify the team here (mail, Slack plugin, etc.)' }
  }
}

Points to notice:

Ephemeral pod agents (Kubernetes plugin): a fresh pod per build, just like ARC runners.
Single quotes in sh '…$TOKEN…' let the shell expand the secret. Groovy string interpolation ("${TOKEN}") puts the secret into the command line and triggers Jenkins' interpolation warning.
A shared library (@Library) holds common steps, versioned with Git tags.

Running the controller well

Practice	Why
0 executors on the built-in node	Builds never run next to the controller's secrets
Configuration as Code (JCasC) in Git	Rebuild the controller's config from YAML
Pinned plugins in a custom image (`plugins.txt`, `jenkins-plugin-cli`)	Reproducible upgrades; test on staging first
Fewer plugins	Each plugin is attack surface and upgrade risk
Backups of `JENKINS_HOME`	Jobs, build history, credentials
Jenkinsfiles in repos (not UI-configured jobs)	Pipelines are reviewed and versioned
SSO + role-based authorisation	No shared admin accounts

Keep up with security advisories: Jenkins core and plugins publish them regularly, and exposed, unpatched controllers are a common attack target.

Scenario: the CI server is down mid-release

Release day. The Jenkins controller's disk fills up and it won't start; a plugin update last week also left it in a fragile state. The release is blocked, and nobody knows how to build the artifacts without Jenkins.

What should the design have included?

Rebuildable controller: JCasC + pinned plugin image + JENKINS_HOME backups, with the restore tested.
Monitoring of the controller: disk, heap, queue length, executor availability.
Builds that don't depend on CI magic: the build logic lives in scripts or Makefiles in the repo, so the same commands can run elsewhere (another CI system or a controlled manual path) in an emergency.
Signed, reproducible releases (lesson 05), so an emergency build is still verifiable.
Change control for plugin upgrades: staging first, with a rollback plan.

Migrating gradually

Inventory jobs: active vs dead, freestyle vs pipeline, plugin dependencies.
Move build logic into repo scripts first; then the CI system becomes a thin wrapper and is easy to switch.
Migrate team by team (GitHub Actions, GitLab CI, Forgejo Actions…). Tools such as GitHub Actions Importer can convert a first draft of pipelines, which you then review.
Run old and new in parallel for a period, then retire jobs and plugins.
Keep Jenkins healthy until the end: migrations take longer than expected.

Try it: Jenkins as code (lab)

Run Jenkins in Docker or on a kind cluster (the official Helm chart) with JCasC configuring a user, the Kubernetes cloud, and 0 executors on the built-in node.
Create a multibranch pipeline for a repo containing the Jenkinsfile above (adapt the image).
Add a string credential and use it with withCredentials; then try "${TOKEN}" interpolation and read the warning.
Write a tiny shared library with one step (sayHello), tag it v1, and load it with @Library.
Delete the Jenkins container and recreate it from the same configuration; measure how long recovery takes.

Going deeper: Jenkins at scale

Split large installations into several controllers (per business unit) to reduce blast radius and plugin conflicts.
Use agents with the minimum credentials they need; prefer OIDC or short-lived tokens where plugins support them.
Measure queue time and build duration; slow CI is a productivity problem as much as an infrastructure one.

Recap

Declarative Jenkinsfiles in repos, ephemeral Kubernetes agents, shared libraries for reuse.
Handle secrets with credentials binding and single-quoted shell expansion.
Controller hygiene: 0 executors, JCasC, pinned plugins, backups, SSO, and patching.
Design CI so it isn't a single point of failure, and migrate gradually by moving logic into repo scripts first.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.