376 lines
9.1 KiB
Markdown
376 lines
9.1 KiB
Markdown
# Mitigation Plan: [Incident Title]
|
|
|
|
**Date**: YYYY-MM-DD HH:MM UTC
|
|
**Incident**: [Brief description]
|
|
**Root Cause**: [Root cause if known, or "Under investigation"]
|
|
**Severity**: SEV1 / SEV2 / SEV3
|
|
**Created By**: [Name]
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
**Problem**: [What's broken in one sentence]
|
|
|
|
**Impact**: [Who's affected and how]
|
|
|
|
**Solution**: [High-level approach]
|
|
|
|
**ETA**: [Estimated time to resolution]
|
|
|
|
**Example**:
|
|
```
|
|
Problem: Database connection pool exhausted due to connection leak
|
|
Impact: All users unable to access application (100% downtime)
|
|
Solution: Restart application + fix connection leak in code
|
|
ETA: 30 minutes (service restored in 5 min, permanent fix in 30 min)
|
|
```
|
|
|
|
---
|
|
|
|
## Three-Horizon Mitigation
|
|
|
|
### Immediate (Now - 5 minutes)
|
|
|
|
**Goal**: Stop the bleeding, restore service immediately
|
|
|
|
**Actions**:
|
|
- [ ] [Action 1]
|
|
- **What**: [Detailed description]
|
|
- **How**: [Commands/steps]
|
|
- **Impact**: [Expected improvement]
|
|
- **Risk**: [Low/Medium/High + explanation]
|
|
- **Rollback**: [How to undo if it fails]
|
|
- **ETA**: [Time to execute]
|
|
- **Owner**: [Who will do this]
|
|
|
|
**Example**:
|
|
```
|
|
- [ ] Restart payment service to release connections
|
|
- What: Restart payment service to release database connections
|
|
- How: `systemctl restart payment-service`
|
|
- Impact: All 100 connections released, service restored
|
|
- Risk: Low (stateless service, graceful restart)
|
|
- Rollback: N/A (restart is safe)
|
|
- ETA: 2 minutes
|
|
- Owner: Jane (SRE)
|
|
|
|
- [ ] Monitor connection pool for 5 minutes
|
|
- What: Verify connections stay below 80%
|
|
- How: `watch -n 5 'psql -c "SELECT count(*) FROM pg_stat_activity"'`
|
|
- Impact: Early detection if issue recurs
|
|
- Risk: None (monitoring only)
|
|
- Rollback: N/A
|
|
- ETA: 5 minutes
|
|
- Owner: Jane (SRE)
|
|
```
|
|
|
|
**Success Criteria**:
|
|
- [ ] Service health check passing
|
|
- [ ] Users able to access application
|
|
- [ ] Connection pool <80% of max
|
|
- [ ] No active alerts
|
|
|
|
---
|
|
|
|
### Short-term (5 minutes - 1 hour)
|
|
|
|
**Goal**: Tactical fix to prevent immediate recurrence
|
|
|
|
**Actions**:
|
|
- [ ] [Action 1]
|
|
- **What**: [Detailed description]
|
|
- **How**: [Commands/steps]
|
|
- **Impact**: [Expected improvement]
|
|
- **Risk**: [Low/Medium/High + explanation]
|
|
- **Rollback**: [How to undo if it fails]
|
|
- **ETA**: [Time to execute]
|
|
- **Owner**: [Who will do this]
|
|
|
|
**Example**:
|
|
```
|
|
- [ ] Fix connection leak in payment service code
|
|
- What: Add `finally` block to close connection in error path
|
|
- How: Deploy hotfix branch `fix/connection-leak`
|
|
- Impact: Connections properly closed, no leak
|
|
- Risk: Medium (code change requires testing)
|
|
- Rollback: `git revert <commit>` + redeploy
|
|
- ETA: 30 minutes (test + deploy)
|
|
- Owner: Mike (Developer)
|
|
|
|
- [ ] Increase connection pool size
|
|
- What: Increase max_connections from 100 to 200
|
|
- How: ALTER SYSTEM SET max_connections = 200; SELECT pg_reload_conf();
|
|
- Impact: More headroom for traffic spikes
|
|
- Risk: Low (more connections = more memory, but server has capacity)
|
|
- Rollback: ALTER SYSTEM SET max_connections = 100; SELECT pg_reload_conf();
|
|
- ETA: 5 minutes
|
|
- Owner: Tom (DBA)
|
|
|
|
- [ ] Add connection pool monitoring alert
|
|
- What: Alert when connections >80% of max
|
|
- How: Create CloudWatch/Grafana alert
|
|
- Impact: Early warning before exhaustion
|
|
- Risk: None (monitoring only)
|
|
- Rollback: Disable alert
|
|
- ETA: 15 minutes
|
|
- Owner: Jane (SRE)
|
|
```
|
|
|
|
**Success Criteria**:
|
|
- [ ] Code fix deployed and verified
|
|
- [ ] Connection pool increased
|
|
- [ ] Monitoring alert configured
|
|
- [ ] No recurrence in 1 hour
|
|
- [ ] Load test passed (if applicable)
|
|
|
|
---
|
|
|
|
### Long-term (1 hour - days/weeks)
|
|
|
|
**Goal**: Permanent fix and prevention
|
|
|
|
**Actions**:
|
|
- [ ] [Action 1]
|
|
- **What**: [Detailed description]
|
|
- **Priority**: P1 / P2 / P3
|
|
- **Due Date**: [YYYY-MM-DD]
|
|
- **Owner**: [Who will do this]
|
|
|
|
**Example**:
|
|
```
|
|
- [ ] Add automated test for connection cleanup
|
|
- What: Integration test that verifies connections are closed in error paths
|
|
- Priority: P1
|
|
- Due Date: 2025-10-27
|
|
- Owner: Lisa (QA)
|
|
|
|
- [ ] Add connection timeout configuration
|
|
- What: Set connection_timeout = 30s in database config
|
|
- Priority: P2
|
|
- Due Date: 2025-10-28
|
|
- Owner: Tom (DBA)
|
|
|
|
- [ ] Review all database queries for connection leaks
|
|
- What: Audit all DB queries to ensure proper cleanup
|
|
- Priority: P3
|
|
- Due Date: 2025-11-02
|
|
- Owner: Mike (Developer)
|
|
|
|
- [ ] Load test for high-traffic events
|
|
- What: Load test with 10x normal traffic to find bottlenecks
|
|
- Priority: P3
|
|
- Due Date: 2025-11-10
|
|
- Owner: John (DevOps)
|
|
|
|
- [ ] Update runbook with new findings
|
|
- What: Document connection leak troubleshooting steps
|
|
- Priority: P3
|
|
- Due Date: 2025-10-28
|
|
- Owner: Jane (SRE)
|
|
```
|
|
|
|
**Success Criteria**:
|
|
- [ ] All P1 actions completed
|
|
- [ ] Regression test added (prevents future occurrences)
|
|
- [ ] Monitoring improved (detect earlier)
|
|
- [ ] Runbook updated
|
|
- [ ] Post-mortem published
|
|
|
|
---
|
|
|
|
## Risk Assessment
|
|
|
|
### Risks of Mitigation Actions
|
|
|
|
| Action | Risk Level | Risk Description | Mitigation |
|
|
|--------|------------|------------------|------------|
|
|
| [Action 1] | Low/Med/High | [What could go wrong] | [How to reduce risk] |
|
|
|
|
**Example**:
|
|
```
|
|
| Restart service | Low | Brief downtime (5s) | Use graceful restart, off-peak time |
|
|
| Deploy code fix | Medium | Bug in fix could worsen issue | Test in staging first, have rollback ready |
|
|
| Increase connection pool | Low | More memory usage | Server has capacity, monitor memory |
|
|
```
|
|
|
|
### Risks of NOT Mitigating
|
|
|
|
| Risk | Impact | Probability |
|
|
|------|--------|-------------|
|
|
| [Risk 1] | [Impact if we do nothing] | High/Med/Low |
|
|
|
|
**Example**:
|
|
```
|
|
| Service remains down | All users affected, revenue loss | High (will recur) |
|
|
| Connection leak worsens | Database crashes | High |
|
|
| SLA breach | Customer refunds, reputation damage | Medium |
|
|
```
|
|
|
|
---
|
|
|
|
## Communication Plan
|
|
|
|
### Internal Communication
|
|
|
|
**Incident Channel**: #incident-YYYYMMDD-title
|
|
|
|
**Update Frequency**: Every [X] minutes
|
|
|
|
**Stakeholders to Notify**:
|
|
- [ ] Engineering team (#engineering)
|
|
- [ ] Customer support (#support)
|
|
- [ ] Management (#management)
|
|
- [ ] [Other teams]
|
|
|
|
**Update Template**:
|
|
```markdown
|
|
[HH:MM] Update:
|
|
- Status: [Investigating / Mitigating / Resolved]
|
|
- Root Cause: [Known / Under investigation]
|
|
- Current Action: [What we're doing now]
|
|
- Next Steps: [What's next]
|
|
- ETA: [Estimated resolution time]
|
|
```
|
|
|
|
---
|
|
|
|
### External Communication
|
|
|
|
**Status Page**: [URL]
|
|
|
|
**Update Frequency**: Every [X] minutes or when status changes
|
|
|
|
**Status Page Template**:
|
|
```markdown
|
|
[HH:MM] Investigating: We are currently investigating [issue description]. Our team is actively working on a resolution.
|
|
|
|
[HH:MM] Identified: We have identified the issue as [root cause]. We are implementing a fix. ETA: [time].
|
|
|
|
[HH:MM] Monitoring: The fix has been deployed. We are monitoring to ensure stability.
|
|
|
|
[HH:MM] Resolved: The issue has been fully resolved. All services are operating normally. We apologize for the inconvenience.
|
|
```
|
|
|
|
**Customer Email** (if needed):
|
|
- [ ] Draft email
|
|
- [ ] Approve with management
|
|
- [ ] Send to affected customers
|
|
|
|
---
|
|
|
|
## Validation
|
|
|
|
### Before Declaring Resolved
|
|
|
|
Verify all of the following:
|
|
|
|
- [ ] Root cause identified
|
|
- [ ] Immediate fix deployed and verified
|
|
- [ ] Service health check passing for >30 minutes
|
|
- [ ] Users able to access application
|
|
- [ ] Metrics returned to normal (response time, error rate, etc.)
|
|
- [ ] No active alerts
|
|
- [ ] Load test passed (if applicable)
|
|
- [ ] Customer support confirms no ongoing issues
|
|
|
|
### Monitoring After Resolution
|
|
|
|
Monitor for [X] hours after declaring resolved:
|
|
|
|
- [ ] [Metric 1] within normal range
|
|
- [ ] [Metric 2] within normal range
|
|
- [ ] [Metric 3] within normal range
|
|
- [ ] No error spikes
|
|
- [ ] No user complaints
|
|
|
|
**Example**:
|
|
```
|
|
- [ ] Connection pool <50% of max
|
|
- [ ] API response time <200ms (p95)
|
|
- [ ] Error rate <0.1%
|
|
- [ ] Database CPU <70%
|
|
```
|
|
|
|
---
|
|
|
|
## Rollback Plan
|
|
|
|
If mitigation actions fail or make things worse:
|
|
|
|
### Immediate Rollback
|
|
|
|
```bash
|
|
# Rollback code deployment
|
|
git revert <commit>
|
|
npm run deploy
|
|
|
|
# Rollback database config
|
|
ALTER SYSTEM SET max_connections = 100;
|
|
SELECT pg_reload_conf();
|
|
|
|
# Verify rollback
|
|
curl http://localhost/health
|
|
```
|
|
|
|
### When to Rollback
|
|
|
|
Rollback if:
|
|
- [ ] Issue worsens after mitigation
|
|
- [ ] New errors appear
|
|
- [ ] Service remains down >X minutes after mitigation
|
|
- [ ] Metrics worsen (response time, error rate)
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
After incident is resolved:
|
|
|
|
1. [ ] Create post-mortem (within 24 hours)
|
|
- Owner: [Name]
|
|
- Due: [Date]
|
|
|
|
2. [ ] Schedule post-mortem review meeting
|
|
- Date: [Date]
|
|
- Attendees: [List]
|
|
|
|
3. [ ] Track action items to completion
|
|
- Use: [JIRA/GitHub/etc.]
|
|
- Review: Weekly in team meeting
|
|
|
|
4. [ ] Update runbooks based on learnings
|
|
- Owner: [Name]
|
|
- Due: [Date]
|
|
|
|
5. [ ] Share learnings with organization
|
|
- Format: All-hands presentation / Email / Wiki
|
|
- Owner: [Name]
|
|
- Due: [Date]
|
|
|
|
---
|
|
|
|
## Appendix
|
|
|
|
### Commands Reference
|
|
|
|
```bash
|
|
# Useful commands for this incident
|
|
<command1>
|
|
<command2>
|
|
<command3>
|
|
```
|
|
|
|
### Links
|
|
|
|
- **Monitoring Dashboard**: [URL]
|
|
- **Runbook**: [URL]
|
|
- **Related Incidents**: [URL]
|
|
- **Incident Channel**: [Slack/Teams URL]
|
|
|
|
---
|
|
|
|
**Plan Created**: YYYY-MM-DD HH:MM UTC
|
|
**Plan Updated**: YYYY-MM-DD HH:MM UTC
|
|
**Status**: Active / Executed / Superseded
|