Skip to content

The 3am Incident That Followed The Playbook

3:17am. The pager vibrates on the nightstand. Half asleep, hand fumbles for phone. The message is three lines. Pod restart storms. API latency spiking. Customers seeing timeouts.

The engineer's first thought isn't "oh god, what now." It's automatic: "open the runbook."

Muscle memory takes over. Hands pull up a laptop still warm from yesterday. The playbook is right there: decision tree, diagnostic steps, escalation paths. No thinking required. Just follow the checklist.

Twenty-three minutes later, the incident is closed. Every step documented. The postmortem writes itself.

This is what happens when you stop improvising and start automating response.

The Problem With 3am Thinking

Before playbooks, incidents were panic, improvisation, and cascading delays:

  • 3:17am: Alert fires. "Is the API actually down or is it monitoring?"
  • 3:20am: Hunt logs. Check cluster health (wrong approach).
  • 3:30am: Escalate to half-asleep on-call engineer.
  • 3:45am: Finally identify the issue.
  • 4:15am: Fix applied, watching.
  • 5:00am: Three people wide awake, can't sleep.
  • 9:00am: Postmortem with spotty memory.

The pattern was consistent. Nobody was stupid. The system just had no structure.

The Real Cost

Every minute of unguided incident response costs you: customer confidence, team morale, and brain cells burning through cortisol you need for the next 6 hours.

The Playbook Structure

A real incident playbook isn't a document. It's a decision tree that removes thinking from 3am decisions.

Incident Alert
├── Severity Classification
│   ├── SEV-1: User-facing outage
│   ├── SEV-2: Degraded performance
│   └── SEV-3: Single service issue
│
├── Immediate Actions (Do this first, no debate)
│   ├── Start the incident thread
│   ├── Notify on-call leadership
│   ├── Enable verbose logging
│   └── Open the runbook
│
├── Diagnosis Phase (Follow this order)
│   ├── Check dashboard (Is it real?)
│   ├── Check metrics (What's the pattern?)
│   ├── Check logs (When did it start?)
│   ├── Check recent changes (What touched it?)
│   └── Form hypothesis
│
├── Response Phase (Execute, don't improvise)
│   ├── Does the fix exist in the playbook?
│   │   ├── Yes → Execute step-by-step
│   │   └── No → Escalate, but document what you try
│   └── Monitor recovery metrics
│
└── Resolution (Close deliberately)
    ├── Verify metrics return to baseline
    ├── Run rollback if fix wasn't stable
    ├── Document exactly what happened
    └── Schedule postmortem

The key insight: every decision point is pre-thought. No creative problem-solving at 3am. Just execution.

What The Playbook Should Include

1. Severity Classifier

Map symptoms to severity levels:

Alert Severity First Action Escalate If
API error rate > 5% SEV-1 Page on-call lead Error rate doesn't drop in 5 min
Pod restart storms SEV-1 Check node health Persists after rolling restart
DB connection pool exhausted SEV-2 Check active queries Can't identify query after 2 min
Memory spike but stable SEV-3 Monitor 10 minutes Goes beyond baseline + 20%

Template: "If you see X and it matches Y, it's SEV-Z. First action is..."

2. Diagnostic Checklist

Pre-written commands that answer "what's actually happening":

# Pod restart storms: check node status
kubectl get nodes -o wide
kubectl describe node <unhealthy-node>

# API latency spike: check goroutine count
curl http://service:9090/metrics | grep goroutine

# Database issue: check connection pool
SELECT count(*) FROM pg_stat_activity;

# Recent deployment: check rollout status
kubectl rollout history deployment/api --revision=<latest>
kubectl rollout undo deployment/api --to-revision=<previous>

No room for interpretation. No "let me think about what might help." Just execute.

3. Decision Logic

For each symptom, define the decision path:

If: Pod restart storms + Node in NotReady state
Then:
  1. Don't try to debug the pod
  2. Cordon the node immediately
  3. Page infrastructure team
  4. Scale up 2 additional nodes
  5. Wait for pods to migrate
  6. Proceed to recovery check

No branching on "but what if." Just "if condition matches, do this."

Pre-Decision Wins

The most valuable decisions are the ones made before the incident. Every "if-then" rule written at 2pm saves 5 minutes of confused thinking at 3am.

Link to fix playbooks:

5. Escalation Thresholds

Define when to call for help. Removes ego from the decision:

  • 2 minutes of unidentified issue
  • Any SEV-1 that hasn't improved in 3 minutes
  • Any action that requires code change
  • Any database operation beyond SELECT

The 3am Reality

What actually happened:

3:17am: Alert fires. Engineer wakes up.

3:18am: Opens incident channel. "Acknowledged, reading playbook."

3:19am: Classification: "Pod restart storms." SEV-1.

3:20am: Diagnostics: kubectl get nodes. One node NotReady.

3:21am: Decision tree: Cordon node, scale up.

kubectl cordon node-07
kubectl scale deployment/api --replicas=12

3:23am: Pods migrate. Error rate returns to baseline.

3:25am: Postmortem outline filled from incident channel.

3:26am: Incident closed. Back to bed.

Nine minutes of focused action. No improvisation. No second-guessing.

Muscle Memory vs. Thinking

The difference between this 23-minute incident and the old 90-minute ones wasn't intelligence. It was structure.

At 3am, your brain is:

  • 40% slower at complex reasoning
  • Prone to fixation on first hypothesis
  • Terrible at context-switching
  • Vulnerable to panic spirals

But your brain is also:

  • Excellent at pattern recognition
  • Capable of executing sequences perfectly
  • Able to follow directions even while drowsy
  • Reliable at checklist completion

The playbook is designed for 3am brains, not fresh 9am brains. It doesn't ask you to think. It gives you steps.

The Practice

Run the playbook during business hours, during drills, during load tests. Make it muscle memory. When the 3am alert hits, execution is automatic.

Tools That Enforce The Playbook

1. Runbook Repository

runbooks/
├── kubernetes/node-recovery.md
├── postgres/connection-pool.md
└── observability/alert-validation.md

Stored in git. Version controlled. Updated in postmortems.

2. Incident Template

Standard Slack template:

SEV-1 | Pod Restart Storms | Started 3:17am
Symptoms: Error rate 12%, pod restart loop
Actions: Cordoned node-07, scaled cluster up
Result: Stable at 3:26am

3. Alert to Runbook Linking

alert: PodRestartStorm
annotations:
  runbook_url: https://runbooks.example.com/kubernetes/pod-crashloop.md

Click the alert. Runbook opens. Steps are waiting.

The Postmortem Writes Itself

Because the incident was documented live, the postmortem is nearly written. The meeting isn't "figure out what happened." It's "understand why the warning signs were missed."

# Incident: Pod Restart Storms - 2026-01-08

Timeline: 3:17am alert, 3:20am node identified, 3:26am closed

Root Cause: Kubelet memory leak after 45-day uptime

Fixes:
1. Add kubelet memory alerting
2. Enable node auto-restart after 40 days
3. Add kubelet memory to pre-incident checks

Impact: 9 minutes, ~50 affected requests

This took 5 minutes to write. Complete. Actionable.

Building Your Playbook

Step 1: Catalog every SEV-1 from the last 2 years. What was the issue? How long did it take? What was the decision path?

Step 2: Extract the decision tree. For each incident type:

  • Symptom classification
  • First action
  • Diagnosis sequence
  • Decision rules (if X, do Y)

Step 3: Write runbooks with exact commands and expected outputs.

Step 4: Wire it into alerts. Every alert has a runbook_url.

Step 5: Practice monthly. Follow the exact playbook. Deviations signal updates needed.

Step 6: Update after incidents. "What would the playbook say?" If the answer is "nothing," add it.

The Shift In Thinking

Before playbooks: "I hope nothing breaks because I don't know what I'll do."

After playbooks: "Something will break. I know exactly what I'll do."

That confidence is worth more than any alerting system.

The engineer at 3:17am isn't panicked, improvising, or waking up the team with vague questions. They're executing a proven sequence. They've practiced it.

Twenty-three minutes. Playbook followed. Incident resolved. Back to sleep.

Further Reading

Full incident response patterns and templates:

The best response to a 3am incident is the one you don't have to think about.

Comments