Last-Write-Wins & Clock Skew.

❌The myth

"Two writers hit the same row. Whoever has the newer updated_at wins. Timestamps are objective — it's just physics."

✅Actually

updated_at doesn't measure when something happened. It measures what some machine's clock said when the code ran. Those are different numbers, and on a fleet of nodes they disagree by milliseconds — sometimes seconds.

NTP-synced servers in one datacenter typically drift 1–10ms apart. A VM that just resumed from a pause, or one whose NTP daemon died quietly three weeks ago, can be hundreds of ms or minutes off. Nothing alerts you, because every node is individually confident.

Last-Write-Wins (LWW) doesn't resolve the conflict. It picks a winner using a number you don't control, and silently throws the loser's data away. No error, no conflict row, no log line.

🔥The 90-second proof

Two app servers, A and B. A's clock is 400ms fast.

t=0ms   User edits profile on A → writes bio="Engineer",  updated_at=1000400
t=250ms User edits again on B  → writes bio="Sr Engineer", updated_at=1000250

Reconciliation (CRDT merge, multi-region replica, or a plain WHERE updated_at < ?) compares 1000250 < 1000400 and keeps "Engineer".

The user's LAST edit lost to their FIRST one. They refresh, see the old bio, and file a bug you will never reproduce — because the race needs A's drift plus a sub-second double-save.

🧭Where this actually bites

• Multi-region active-active (DynamoDB global tables, Cassandra, Cosmos) — LWW is the default merge
• Offline-first mobile sync where the DEVICE supplies the timestamp (device clocks are user-settable — someone's phone is in 2031)
• UPDATE ... WHERE updated_at < $new as a homegrown guard
• Cassandra deletes: a tombstone with a future timestamp makes a row un-writable until real time catches up

🛠️What to use instead

1. Monotonic version counter — UPDATE ... SET bio=?, version=version+1 WHERE id=? AND version=?, 0 rows updated = conflict, retry. Single-primary, cheap, correct. Default to this.
2. Logical clocks — Lamport counters or vector clocks. Order by causality, not wall time. Detects "these two are genuinely concurrent" instead of guessing.
3. Hybrid Logical Clocks (HLC) — physical time + logical counter in one 64-bit value. Stays human-readable AND keeps causality. What CockroachDB and YugabyteDB use.
4. Bounded uncertainty — Spanner's TrueTime reports an interval, not an instant, and waits out the error bar before committing. Correct because it treats clock error as a first-class number.
5. Don't merge at all — store both and let a human (or app rule) decide. Right answer for shopping carts: union the items.

⚠️The one practitioners get wrong

"We'll just use the DATABASE's clock, not the app server's — NOW() on the server side." Better, and it kills a real class of bugs. But NOW() in Postgres returns transaction START time, so two overlapping transactions can both stamp the same instant — and the moment you add a replica or a second region, you're back to comparing two machines' clocks. It narrows the window; it doesn't close it.

Rule of thumb: timestamps are for humans reading logs. Versions are for machines resolving conflicts.

🎯Your exercise

Grep your codebase for updated_at <, updated_at >, or ORDER BY updated_at DESC LIMIT 1 used to pick a winner (not just to display). For each hit, answer: if the clock that wrote that row were 500ms fast, what data silently disappears?

Then pick the worst one and convert it to a version column. Bonus: run chronyc tracking (or timedatectl) on two of your prod boxes and compare the offsets — tell me how far apart they actually are.