AWS VMs → kubeadm → Cilium → Jenkins deploys from Git

The classic interview build, end to end: create EC2 instances on AWS, bootstrap a kubeadm cluster, bring up Cilium, install Jenkins, and make a git push trigger a pipeline that builds an image, pushes it to a registry and deploys it to the cluster, then prove it works and explain every decision.

The build: push to GitHub → Jenkins builds and pushes the image → deploys to a kubeadm + Cilium cluster on EC2.

The brief

"Create VMs on AWS, bootstrap a Kubernetes cluster with kubeadm, use Cilium as the CNI, and set up Jenkins so that a push to Git triggers a job that builds and deploys the application. Show me how you'd check it works, and tell me where the image is stored."

This project builds exactly that, and gives you the talking points for every step.

It's a toy factory line. You write a new toy design and drop it in the inbox (git push). A bell rings (webhook) and the factory robot (Jenkins) builds the toy, puts it on the warehouse shelf (registry) with a label saying exactly which design it is (commit tag), and tells the shop (the cluster) to put the new toy on display. You walk into the shop to check it's there (curl).

Resources needed

Resource	Detail	Why
AWS account	IAM user/role allowed to manage EC2	Create the VMs
AWS CLI v2 + an SSH key pair	`aws configure`, `aws ec2 create-key-pair`	Script everything
VPC + public subnet	The default VPC is fine for a lab	Network for the VMs
Security group	SSH/6443/NodePorts/8080 from your IP; all traffic inside the group	Firewall
3 × EC2 `t3.medium` (2 vCPU, 4 GiB), Ubuntu 24.04, 20 GiB gp3	`cp1`, `w1`, `w2`	Cluster nodes
1 × EC2 `t3.medium`	`jenkins` (Jenkins + Docker + kubectl)	CI server
GitHub account + repo	`hello-k8s` app repo	Source + webhook
Container registry	GHCR or Docker Hub (or ECR, see decisions)	Store images
Local tools	`git`, `ssh`; `kubectl` optional on your laptop	Operate

Cost: four small instances plus disks cost a few dollars a day on demand (check the EC2 pricing page for your region). Terminate everything when you finish.

The flow you're building

You push a commit to the app repo on GitHub.
GitHub sends a webhook to Jenkins.
Jenkins checks out the code, runs a quick test, builds an image tagged with the commit SHA, and pushes it to the registry.
Jenkins deploys: applies the manifests with the new image tag, using a limited ServiceAccount.
Kubelets pull the image (with an imagePullSecret) and the Deployment rolls out.
You check the app through the NodePort (and in the pipeline logs).

Step 1: network, security group, instances

$ export AWS_REGION=eu-west-1
$ MYIP=$(curl -s https://checkip.amazonaws.com)
$ VPC=$(aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text)
$ SUBNET=$(aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC --query 'Subnets[0].SubnetId' --output text)
$ SG=$(aws ec2 create-security-group --group-name k8s-lab --description "kubeadm lab" --vpc-id $VPC --query GroupId --output text)
$ for port in 22 6443 8080; do aws ec2 authorize-security-group-ingress --group-id $SG --protocol tcp --port $port --cidr $MYIP/32; done
$ aws ec2 authorize-security-group-ingress --group-id $SG --protocol tcp --port 30000-32767 --cidr $MYIP/32
$ aws ec2 authorize-security-group-ingress --group-id $SG --protocol -1 --source-group $SG    # node-to-node
$ aws ec2 create-key-pair --key-name lab-key --query KeyMaterial --output text > lab-key.pem && chmod 600 lab-key.pem
$ AMI=$(aws ssm get-parameters --names /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id --query 'Parameters[0].Value' --output text)
$ for name in cp1 w1 w2 jenkins; do
    aws ec2 run-instances --image-id $AMI --instance-type t3.medium --key-name lab-key \
      --security-group-ids $SG --subnet-id $SUBNET --associate-public-ip-address \
      --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=20,VolumeType=gp3}' \
      --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$name},{Key=Project,Value=k8s-lab}]" >/dev/null
  done
$ aws ec2 describe-instances --filters Name=tag:Project,Values=k8s-lab Name=instance-state-name,Values=running \
    --query 'Reservations[].Instances[].[Tags[?Key==`Name`]|[0].Value,PublicIpAddress,PrivateIpAddress]' --output table

GitHub must reach Jenkins for the webhook: either also allow port 8080 from GitHub's published webhook IP ranges (https://api.github.com/meta, key hooks), or use SCM polling in the lab. Don't open 8080 to the whole internet.

(Terraform is the better long-term way to create all this; see Terraform & Infrastructure as Code.)

Step 2: prepare the three nodes

SSH to each node (ssh -i lab-key.pem ubuntu@<public-ip>), set its hostname, and run the node preparation from Kubernetes Administration, lesson 09 (swap off, kernel modules and sysctls, containerd with SystemdCgroup = true, kubelet/kubeadm/kubectl pinned):

$ sudo hostnamectl set-hostname cp1        # w1 / w2 on the workers
# then: swap off, modules + sysctl, containerd, kube packages (lesson 09, Step 1)

Step 3: bootstrap the cluster with kubeadm

On cp1:

$ CP_PRIVATE_IP=$(hostname -I | awk '{print $1}')
$ CP_PUBLIC_IP=$(curl -s https://checkip.amazonaws.com)
$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
    --apiserver-advertise-address=$CP_PRIVATE_IP \
    --apiserver-cert-extra-sans=$CP_PUBLIC_IP
$ mkdir -p ~/.kube && sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) ~/.kube/config

--apiserver-cert-extra-sans adds the public IP to the API certificate so you can use kubectl from your laptop. (A public IP changes when an instance is stopped; use an Elastic IP if you'll stop and start the lab.)

On w1 and w2, run the kubeadm join … command printed by init. Nodes show NotReady until the CNI is installed.

Step 4: bring up Cilium

On cp1:

$ CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
$ curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
$ sudo tar xzvf cilium-linux-amd64.tar.gz -C /usr/local/bin && rm cilium-linux-amd64.tar.gz
$ cilium install --set ipam.mode=kubernetes
$ cilium status --wait
$ kubectl get nodes
NAME   STATUS   ROLES           AGE   VERSION
cp1    Ready    control-plane   6m    v1.31.x
w1     Ready    <none>          4m    v1.31.x
w2     Ready    <none>          4m    v1.31.x
$ cilium connectivity test          # optional, takes several minutes

ipam.mode=kubernetes makes Cilium use the pod CIDR that kubeadm assigned (10.244.0.0/16), which avoids overlapping your VPC range. Cilium's default VXLAN tunnelling works inside one security group because node-to-node traffic is allowed.

Step 5: decide where the image lives

Registry	Pros	Pull from kubeadm nodes
GHCR (GitHub Container Registry)	Next to your code; free for public images	imagePullSecret with a read-only token (private images)
Docker Hub	Familiar	imagePullSecret; mind pull rate limits
Amazon ECR	In your AWS account, IAM-controlled	Tokens expire after 12 h: use the kubelet ECR credential provider plus an instance IAM role on nodes (EKS does this for you)

This lab uses GHCR with a private image and an imagePullSecret. Say the ECR point in an interview; it shows you understand how kubelets authenticate.

$ kubectl create namespace demo
$ kubectl -n demo create secret docker-registry regcred \
    --docker-server=ghcr.io --docker-username=<github-user> --docker-password=<token-with-read:packages>

Step 6: the app repository

hello-k8s/
├── index.html
├── Dockerfile
├── Jenkinsfile
└── k8s/
    ├── deployment.yaml
    └── service.yaml

# Dockerfile
FROM nginx:1.27-alpine
COPY index.html /usr/share/nginx/html/index.html

# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 2
  selector:
    matchLabels: { app: hello }
  template:
    metadata:
      labels: { app: hello }
    spec:
      imagePullSecrets: [ { name: regcred } ]
      containers:
        - name: web
          image: IMAGE_PLACEHOLDER
          ports: [ { containerPort: 80 } ]
          readinessProbe:
            httpGet: { path: /, port: 80 }
          resources:
            requests: { cpu: 50m, memory: 32Mi }
            limits: { memory: 64Mi }

# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: NodePort
  selector: { app: hello }
  ports: [ { port: 80, targetPort: 80, nodePort: 30080 } ]

Step 7: Jenkins

On the jenkins instance: Docker (to build images), Java 17+, Jenkins from its official repository, and kubectl:

$ sudo apt-get update && sudo apt-get install -y docker.io fontconfig openjdk-17-jre git
$ sudo curl -fsSLo /usr/share/keyrings/jenkins-keyring.asc https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
$ echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/" | sudo tee /etc/apt/sources.list.d/jenkins.list
$ sudo apt-get update && sudo apt-get install -y jenkins
$ sudo usermod -aG docker jenkins && sudo systemctl restart jenkins
$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword

(Check the Jenkins installation page for the current signing key and supported Java versions.) Install kubectl from pkgs.k8s.io (same minor as the cluster). Open http://<jenkins-public-ip>:8080, unlock, install the suggested plugins (they include Git, Pipeline and GitHub integration), and create an admin user.

A limited identity for deployments (run on cp1):

# jenkins-deployer.yaml
apiVersion: v1
kind: ServiceAccount
metadata: { name: jenkins-deployer, namespace: demo }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: jenkins-deployer, namespace: demo }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: edit }
subjects: [ { kind: ServiceAccount, name: jenkins-deployer, namespace: demo } ]
---
apiVersion: v1
kind: Secret
metadata:
  name: jenkins-deployer-token
  namespace: demo
  annotations: { kubernetes.io/service-account.name: jenkins-deployer }
type: kubernetes.io/service-account-token      # long-lived token: fine for a lab, rotate in real life

$ kubectl apply -f jenkins-deployer.yaml
$ TOKEN=$(kubectl -n demo get secret jenkins-deployer-token -o jsonpath='{.data.token}' | base64 -d)
$ kubectl config --kubeconfig=jenkins.kubeconfig set-cluster lab --server=https://$CP_PRIVATE_IP:6443 \
    --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true
$ kubectl config --kubeconfig=jenkins.kubeconfig set-credentials jenkins --token=$TOKEN
$ kubectl config --kubeconfig=jenkins.kubeconfig set-context lab --cluster=lab --user=jenkins --namespace=demo
$ kubectl config --kubeconfig=jenkins.kubeconfig use-context lab

In Jenkins → Manage Jenkins → Credentials, add:

ghcr (Username with password): GitHub user + a token with write:packages.
kubeconfig-demo (Secret file): jenkins.kubeconfig.

Step 8: the pipeline

// Jenkinsfile
pipeline {
  agent any
  environment {
    IMAGE = "ghcr.io/<github-user>/hello-k8s"
  }
  stages {
    stage('Checkout') {
      steps {
        checkout scm
        script { env.TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() }
      }
    }
    stage('Test') {
      steps { sh 'test -s index.html && grep -q "<html" index.html' }
    }
    stage('Build & push') {
      steps {
        withCredentials([usernamePassword(credentialsId: 'ghcr', usernameVariable: 'REG_USER', passwordVariable: 'REG_PASS')]) {
          sh '''
            echo "$REG_PASS" | docker login ghcr.io -u "$REG_USER" --password-stdin
            docker build -t "$IMAGE:$TAG" .
            docker push "$IMAGE:$TAG"
          '''
        }
      }
    }
    stage('Deploy') {
      steps {
        withCredentials([file(credentialsId: 'kubeconfig-demo', variable: 'KUBECONFIG')]) {
          sh '''
            kubectl apply -f k8s/service.yaml
            sed "s|IMAGE_PLACEHOLDER|$IMAGE:$TAG|" k8s/deployment.yaml | kubectl apply -f -
            kubectl rollout status deploy/hello --timeout=120s
          '''
        }
      }
    }
  }
}

Create a Pipeline job → "Pipeline script from SCM" → Git → your repo, branch main → tick GitHub hook trigger for GITScm polling. In GitHub → repo → Settings → Webhooks → add http://<jenkins-public-ip>:8080/github-webhook/ (content type application/json, push events).

Step 9: check the app

$ git commit -am "hello v2" && git push                  # triggers the pipeline
$ kubectl -n demo get pods -o wide                       # new pods, new image
$ kubectl -n demo get deploy hello -o jsonpath='{.spec.template.spec.containers[0].image}'
ghcr.io/<github-user>/hello-k8s:3f9c2a1
$ kubectl -n demo rollout history deploy/hello
$ curl -s http://<w1-public-ip>:30080 | head -5          # your page, new version

Also check: the Jenkins build log (every stage green), the image with that tag in GHCR, and kubectl -n demo describe pod events (pull, start, readiness).

Rollback: git revert the bad commit and push (the pipeline redeploys), or kubectl -n demo rollout undo deploy/hello for an emergency.

When it breaks

Symptom	Check
Nodes `NotReady`	`cilium status`; security group allows node-to-node traffic
Webhook doesn't trigger	Webhook "Recent deliveries" in GitHub; URL ends with `/github-webhook/`; job trigger ticked; 8080 reachable from GitHub
`docker: permission denied` in Jenkins	`jenkins` user in the `docker` group, Jenkins restarted
`ImagePullBackOff`	Tag exists? `regcred` in namespace `demo`? Token has `read:packages`?
`Unauthorized` / `forbidden` from kubectl in Jenkins	Token, RoleBinding, and the server address reachable from the Jenkins VM
NodePort unreachable	Security group rule for 30000–32767 from your IP; pods Ready

Interview talking points

Why commit SHA tags and not latest (unique, traceable, triggers rollouts, clean rollbacks).
Where images live and how nodes authenticate (imagePullSecret vs ECR credential provider; EKS does it for you).
Least privilege for CI (namespace-scoped ServiceAccount), secrets kept in Jenkins credentials, never in Git.
Scaling Jenkins: build agents on Kubernetes instead of building on the controller (see CI/CD & Software Supply Chain, lesson 08).
What you'd improve: HA control plane, Ingress + TLS instead of NodePort, image signing and scanning, and GitOps so CI doesn't hold cluster credentials at all (next project).

Clean up

$ IDS=$(aws ec2 describe-instances --filters Name=tag:Project,Values=k8s-lab --query 'Reservations[].Instances[].InstanceId' --output text)
$ aws ec2 terminate-instances --instance-ids $IDS
$ aws ec2 wait instance-terminated --instance-ids $IDS
$ aws ec2 delete-security-group --group-id $SG && aws ec2 delete-key-pair --key-name lab-key

Command summary

# AWS
aws ec2 create-security-group … / authorize-security-group-ingress … / create-key-pair …
aws ssm get-parameters --names /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id
aws ec2 run-instances --image-id $AMI --instance-type t3.medium … (cp1, w1, w2, jenkins)
# Nodes (all three): swap off, modules/sysctl, containerd, kubelet/kubeadm/kubectl  (Admin lesson 09)
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$CP_PRIVATE_IP --apiserver-cert-extra-sans=$CP_PUBLIC_IP
sudo kubeadm join … (workers)
cilium install --set ipam.mode=kubernetes && cilium status --wait
kubectl create namespace demo
kubectl -n demo create secret docker-registry regcred --docker-server=ghcr.io …
kubectl apply -f jenkins-deployer.yaml   # SA + RoleBinding + token Secret
kubectl config --kubeconfig=jenkins.kubeconfig set-cluster/set-credentials/set-context/use-context …
# Jenkins VM
sudo apt-get install -y docker.io openjdk-17-jre jenkins && sudo usermod -aG docker jenkins
# Pipeline: checkout → test → docker build/push $IMAGE:$TAG → kubectl apply + rollout status
# Check
kubectl -n demo get pods -o wide ; kubectl -n demo rollout history deploy/hello ; curl http://<node-ip>:30080
# Clean up
aws ec2 terminate-instances --instance-ids $IDS ; aws ec2 delete-security-group … ; aws ec2 delete-key-pair …

Recap

Resources: 3 cluster VMs + 1 Jenkins VM, one security group, a GitHub repo, a registry.
Flow: push → webhook → Jenkins (test, build, push image:SHA) → kubectl apply with a limited ServiceAccount → rollout → check via NodePort.
Decisions to explain: SHA tags, registry and pull credentials, least privilege, what you'd improve (HA, Ingress/TLS, signing, GitOps).

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.