Clocks in Distributed Systems: Why You Can’t Trust Timestamps¶
Many design bugs come from assuming that time is the same on every machine. Interviewers like candidates who know the limits.
Physical clocks drift¶
Each server’s clock drifts, and NTP corrects it, typically keeping servers within a few milliseconds on a good network, worse across the internet. NTP can also step the clock backward. Consequences:
- “Last write wins” by timestamp can silently discard the newer write if its server’s clock was behind.
- Measuring durations with wall-clock time can give negative results. Use a monotonic clock for timeouts and latency.
- Lease-based locks break if a node’s clock runs fast or the process pauses (GC, VM migration) past the lease.
Logical clocks¶
- Lamport timestamps: a counter incremented on each event and updated to max(local, received) + 1 on each message. They give an order consistent with causality.
- Vector clocks: one counter per node, detecting concurrent updates.
- Hybrid logical clocks (HLC): combine physical time with a logical counter, close to wall time but still preserving causality. Used by CockroachDB and others.
Bounded uncertainty¶
Google Spanner’s TrueTime exposes time as an interval [earliest, latest] using GPS and atomic clocks. A transaction waits out the uncertainty before committing (“commit wait”), which gives globally consistent ordering.
Practical guidance¶
- Don’t use timestamps from different machines to order events that matter; use a sequence from a single leader, a log offset, or logical clocks.
- Use fencing tokens (increasing numbers) with locks so a stale holder can’t write.
- Store times in UTC and record which clock produced them.
Interview tip¶
When you rely on time, say which clock and what happens under skew. It’s a quick way to show depth.
Disclosure: I help build Linkjob AI (www.linkjob.ai/), an AI assistant for technical interviews.