Prasanta Kumar Behera
Incident Handling — On-Call Playbook & Real Scenarios›13 · Clock drift: NTP sync failed
Learning Hub / Incident Handling / Incident Handling — On-Call Playbook & Real Scenarios

Lesson 13 of 19 · Real-world incident scenarios

Clock drift: NTP sync failed

Random TLS errors ('certificate is not yet valid'), rejected tokens, etcd clock warnings and logs out of order across a cluster: the nodes' clocks have drifted because NTP stopped working. Check chrony, find why time sync broke, correct time safely, and make time a monitored dependency.

Practitioner → Advanced
Key wordsNTPchronyclock driftx509 not yet validtoken expiryetcd clock differenceUDP 123local time sourcenode_timex_offset_seconds

The page

Several teams report odd failures on one cluster: some kubectl calls fail with x509: certificate has expired or is not yet valid, a new node won't join ("certificate is not valid yet"), OIDC logins are rejected intermittently, and etcd logs show prober found high clock drift. A firewall change last week tightened outbound rules at this site.

First five minutes

  • Impact: intermittent authentication and TLS failures; risk to etcd stability if drift keeps growing.
  • Scope: all nodes at one site? One node? Correlate with the firewall change.
  • Compare clocks across nodes right away.

A school where every classroom clock shows a different time. The bell rings at the wrong moment, the "come back at 10" notes make no sense, and the librarian thinks your return slip is from next week. Everyone needs to set their clock from the same master clock, and someone should notice when a clock starts drifting.

Diagnose

$ timedatectl
System clock synchronized: no
              NTP service: active
$ chronyc tracking
Reference ID    : 00000000 ()
Stratum         : 0
System time     : 94.382112 seconds slow of NTP time
$ chronyc sources -v
MS Name/IP address         Stratum Poll Reach LastRx Last sample
^? ntp1.example.net              0   10     0     -     +0ns[   +0ns] +/-    0ns

Reach 0 and ^? mean chrony can't reach its sources. Check firewall rules for UDP 123, DNS resolution of the NTP server names, and whether the configured servers exist at this site.

Symptoms explained

Symptom Why
x509: … not yet valid / expired Node clock behind/ahead of the certificate's validity window
Tokens rejected (OIDC, service account) iat/exp checks fail with skewed clocks
etcd high clock drift warnings Members' clocks differ; leases and elections can misbehave
Logs and traces out of order Correlation across nodes breaks

Fix

  1. Restore reachability to NTP servers (firewall rule for UDP 123), or point chrony at a local time source.
  2. For small offsets, chrony slews gradually once sources are reachable.
  3. For large offsets, step the clock deliberately, one node at a time (drain first if it hosts sensitive workloads):
$ sudo chronyc makestep
$ chronyc tracking | grep 'System time'
  1. After time is right, retry failed operations (node join, certificate issuance) and restart components that got stuck in error states.

Prevent

  • At least two or three time sources, including a local one at each site (router, site server, or GPS/PTP for precise needs).
  • Document and test firewall rules for NTP; include them in site acceptance.
  • Alert on offset (node-exporter's node_timex_offset_seconds, or chrony exporters) and on node_timex_sync_status == 0.
  • Put time in the dependency map as tier 0 (see SRE & Production Incident Response, lesson 06).

Try it: drift a node (lab)

  1. On a lab VM, stop chrony and set the clock back with date -s '-10 minutes' (lab only).
  2. Try kubectl against a cluster or curl https:// a site and observe TLS errors.
  3. Start chrony, block UDP 123 with nftables, and watch chronyc sources show no reach.
  4. Unblock, run chronyc makestep, and confirm sync.
  5. Add an alert rule for abs(node_timex_offset_seconds) > 0.05.

Going deeper: time at scale

  • Virtual machines can drift after host pauses or live migration; keep NTP running in guests even if the hypervisor syncs time.
  • For telecom and some industrial workloads, PTP (IEEE 1588) gives far higher precision than NTP.
  • Keep time zones out of it: servers in UTC, convert at display time.

Recap

  • Odd TLS/token/etcd errors across nodes → check clocks first.
  • chronyc tracking / sources show offset and reachability; UDP 123 blocked is a classic cause.
  • Restore sources, let chrony slew, step carefully for large offsets.
  • Multiple (local) sources, documented firewall rules, and offset alerts.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.