Chasing intermittent 504s across four services

Users hit intermittent gateway timeouts. Each service dashboard looked healthy, because each one only saw its own slice of the request. Without a way to follow one request across all four hops, every investigation restarted from scratch.

The failure was intermittent and crossed four systems owned by different teams, in two languages, on two clouds. Reproducing it on demand was not possible, so the fix had to come from evidence gathered in production.

• Instrumented the chain with OpenTelemetry and threaded a correlation ID through every hop, so a single failing request could be reconstructed end to end
• Used the resulting traces to locate where latency actually accumulated, rather than where it was assumed to
• Tuned HTTP keep-alive behaviour along the chain once the traces showed connection churn was the cost

• Error rate fell from 1.2% to ~0.02%
• A repeatable way to trace any future cross-service failure

The instrumentation should have existed before the incident, not because of it. The investigation took weeks; once the trace was readable, the fix took an afternoon. That ratio is the whole argument for wiring correlation IDs in on day one.

• OpenTelemetry
• Node.js
• React
• Django
• Azure