Distributed Tracing.

🧩The mental model

Picture a relay race. A single user request is the baton. It gets passed from the API gateway → auth service → orders service → payments → database. Logs are like each runner scribbling a note in their own diary. Useful — but nobody has the full race timeline. A trace is the baton itself carrying a shared ID, so afterward you can reconstruct the whole run: who ran when, and who was slow.

Two words to nail down:

• Trace = the entire journey of one request (one shared trace_id).
• Span = one unit of work inside it (a single service call, a DB query). Spans nest: a parent span has children.

🔗The trick that makes it work: context propagation

Nothing is magic. The trace_id is literally passed as an HTTP header from service to service. The W3C standard header is traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  └─ trace_id (32 hex) ──┘ └─ parent span_id ┘ └ flags

Each service reads that header, starts its own span as a child, and passes a NEW traceparent (same trace_id, its own span_id) to the next hop. Break that chain and your trace splits into orphans.

🛠️In practice (pseudo-Python with OpenTelemetry)

# incoming request: extract context from headers
ctx = extract(request.headers)
with tracer.start_as_current_span("orders.create", context=ctx) as span:
    span.set_attribute("order.id", order_id)
    # outgoing call: inject context into headers
    headers = {}
    inject(headers)                 # adds traceparent
    requests.post(PAYMENTS_URL, headers=headers, json=payload)

That extract → span → inject dance is the whole game. Most frameworks auto-instrument HTTP clients so you rarely write it by hand — but knowing it exists is what saves you when a trace mysteriously ends.

⚠️Where intermediate devs get bitten

• Async gaps: sending work to a queue/thread pool often DROPS the context — the worker starts a brand-new trace. You must manually carry the context across the boundary.
• Sampling ≠ logging. Tracing usually samples (e.g. 1–10% of requests) to control cost. That p99 outlier you're chasing may simply not have been sampled. Use tail-based sampling if you need the slow ones.
• Cardinality: don't shove unbounded values (raw emails, full URLs) into span attributes — it explodes your backend's index and bill.
• Clock skew: span timings come from different machines' clocks. A child looking like it started "before" its parent is usually skew, not a bug.

🎯Why you care

When a request is slow, logs tell you WHAT happened in each box; a trace tells you WHERE the time went. A waterfall view showing payments took 1.8s of a 2s request points you at the culprit in seconds instead of grepping five log streams.

🧪Your exercise

Take any two services you have (or two functions across a network call). Generate a trace_id at the entry point and pass it as a header/param all the way through. Then log it as trace_id=... in every service. Now grep one ID across all logs and reconstruct the timeline by hand. Once you feel that pain — and the payoff — reach for OpenTelemetry to automate it.

Question for you: in your current system, name ONE boundary where trace context would silently get dropped. (Hint: look for a queue, a background job, or a new Thread.)