In brief
Fast incident response is not random command execution. The reliable sequence is to define impact, build a timeline, locate the failure boundary, test the strongest hypothesis, and verify recovery with user-facing evidence.
Key takeaways
- Write a one-sentence incident statement before making a change.
- Build a UTC timeline from independent sources.
- Prefer reversible actions that test a specific hypothesis.
- Separate mitigation from root cause and prevention.
Begin with an incident statement
A useful incident statement names the affected service, user impact, scope, start time, and current evidence. For example: production API requests through one region began returning HTTP 502 at 14:08 UTC after deployment 1842, while the default backend hostname remains healthy. This is far more actionable than saying the website is down.
The statement should evolve as evidence changes. Mark assumptions explicitly, and do not promote correlation to cause. A deployment near the incident time is important evidence, but it becomes a likely cause only when logs, configuration comparison, rollback behavior, or a controlled test supports it.
Build the timeline and failure boundary
Normalize logs, alerts, deployment records, configuration activity, and user reports to UTC. Identify the last known good transaction and first known failure. A tight timeline exposes changes that generic dashboards hide and makes it easier to compare evidence from Azure Activity Log, application telemetry, gateways, databases, and delivery systems.
Then walk the request path from client to dependency. Test DNS, edge routing, gateway, application worker, identity, data service, and external dependency separately. The goal is not to prove every layer healthy; it is to find the first layer where expected behavior changes.
- Use the same hostname, route, identity, and region as the failed request.
- Compare healthy and unhealthy instances or environments.
- Preserve correlation IDs and timestamps across service boundaries.
- Record negative evidence, because a healthy layer narrows the search.
Mitigate with a testable action
Choose the smallest reversible change that addresses the leading hypothesis. A rollback tests a release hypothesis. Removing one unhealthy instance tests an instance hypothesis. Correcting a probe hostname tests a routing hypothesis. Changing several settings at once may restore service, but it destroys the ability to know which change mattered.
When impact requires an immediate broad mitigation, record the exact action and time, then preserve remaining evidence. A restart can be valid, but it should not automatically become the root cause. Treat recovery after restart as evidence of a stateful or worker-level condition that still requires investigation.
Verify recovery and write a useful review
Verify from the user path with a representative transaction, then confirm error rate, latency, dependency health, and every relevant instance. Continue observing through a meaningful workload period. Recovery is a state supported by evidence, not the absence of one alert.
A useful post-incident review separates trigger, root cause, contributing conditions, detection gaps, mitigation, and prevention. Assign follow-up work to controls that would prevent recurrence or reduce time to detection and recovery. Avoid action items that only ask people to be more careful.