Checkpoint: a day on a server · Prasanta Kumar Behera

Put Level 1 to work on a server that has been broken on purpose: a web server that won't start, a page that returns 403, logs a developer can't read, and a disk to tidy. Solve each from the symptoms.

The situation

You're on call. A colleague hands you a server with three complaints and a warning:

"The website is down."
"When it was up, it said 403 Forbidden."
"Dev can't read the app logs in /var/log/myapp."
"Also, check what's using the most space in /var/log."

It's like being the new caretaker of a school on day one: the front door is jammed, a classroom is locked, a teacher can't get into the staff room, and a cupboard is overflowing. You don't guess. You check each one, find the real reason, and fix only that.

Set up the broken server

Use a disposable Ubuntu VM (not a machine you care about). Save this as checkpoint-setup.sh and run it with sudo bash checkpoint-setup.sh:

#!/usr/bin/env bash
# Breaks a fresh Ubuntu VM on purpose, for the Linux Level 1 checkpoint.
set -euo pipefail
apt-get update -qq && apt-get install -y -qq nginx > /dev/null

# 3) app logs readable only by the 'applog' group
groupadd -f applog
mkdir -p /var/log/myapp
for i in 1 2 3; do echo "$(date) INFO request ok" >> /var/log/myapp/app.log; done
echo "$(date) ERROR database connection refused" >> /var/log/myapp/app.log
chown root:applog /var/log/myapp /var/log/myapp/app.log
chmod 750 /var/log/myapp
chmod 640 /var/log/myapp/app.log

# 2) a page the web server can't read
echo '<h1>Checkpoint</h1>' > /var/www/html/index.html
chmod 600 /var/www/html/index.html

# 1) a missing semicolon in the nginx config
sed -i 's/listen 80 default_server;/listen 80 default_server/' /etc/nginx/sites-available/default
systemctl restart nginx || true
echo "Server broken on purpose. Good luck!"

Also create a developer account to test with: sudo adduser dev.

Now stop reading and try each problem yourself. Solutions follow.

Problem 1: the website is down

Your turn

Find out whether nginx is running, why not, and fix it.

Solution

$ systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
     Active: failed (Result: exit-code) since …
$ sudo nginx -t
nginx: [emerg] invalid parameter "listen" in /etc/nginx/sites-enabled/default:23
nginx: configuration file /etc/nginx/nginx.conf test failed

The line before the reported one is missing its ;, so nginx read the next listen as a parameter. (Line numbers may differ slightly.) Open the file with sudo nano /etc/nginx/sites-available/default, add the semicolon back to listen 80 default_server, then:

$ sudo nginx -t && sudo systemctl restart nginx
nginx: configuration file /etc/nginx/nginx.conf test is successful
$ systemctl is-active nginx
active

Habit: always run the config test before restarting a service.

Problem 2: 403 Forbidden

Your turn

curl -I http://localhost now answers, but with a 403. Why, and what's the smallest correct fix?

Solution

$ curl -I http://localhost
HTTP/1.1 403 Forbidden
$ sudo tail -3 /var/log/nginx/error.log
… open() "/var/www/html/index.html" failed (13: Permission denied) …
$ ls -l /var/www/html/index.html
-rw------- 1 root root 20 Sep 27 10:02 /var/www/html/index.html
$ ps -o user= -C nginx | sort | uniq -c
      1 root
      2 www-data

The workers run as www-data, but only root can read the file. Make it world-readable (web pages are public anyway):

$ sudo chmod 644 /var/www/html/index.html
$ curl -s http://localhost
<h1>Checkpoint</h1>

Error code 13 is EACCES (permission denied). The error log told you exactly which file.

Problem 3: dev can't read the app logs

Your turn

sudo -u dev cat /var/log/myapp/app.log fails. Give dev read access without loosening permissions for everyone.

Solution

$ ls -ld /var/log/myapp /var/log/myapp/app.log
drwxr-x--- 2 root applog 4096 … /var/log/myapp
-rw-r----- 1 root applog  246 … /var/log/myapp/app.log
$ sudo usermod -aG applog dev

The directory and file are group-readable by applog, so adding dev to that group is the least-privilege fix. Dev must log in again for the new group to apply. Then:

$ sudo -iu dev
$ grep ERROR /var/log/myapp/app.log
… ERROR database connection refused

Problem 4: what's using the space?

Your turn

List the five largest items under /var/log, and confirm which process is listening on port 80.

Solution

$ sudo du -ah /var/log | sort -rh | head -5
$ sudo ss -ltnp | grep ':80 '
LISTEN 0  511  0.0.0.0:80  0.0.0.0:*  users:(("nginx",pid=2231,fd=6),("nginx",pid=2230,fd=6))

Don't delete logs by hand while a service writes to them. Use the log rotation (logrotate) or journalctl --vacuum-size= for the journal (lesson 07).

Going deeper: write it up

For each problem, write three lines: symptom → evidence → fix. This is the start of a proper incident habit (see SRE & Production Incident Response), and it's exactly what interviewers ask for: "tell me about a time something was broken."

You've finished Linux Level 1

You can navigate, read and search, manage permissions, processes, services and packages, and fix common problems from evidence. Level 2 is troubleshooting in depth: systemd and the journal, disks, host networking, performance and SSH.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.