# Mitigation Plan: [Incident Title]

**Date**: YYYY-MM-DD HH:MM UTC
**Incident**: [Brief description]
**Root Cause**: [Root cause if known, or "Under investigation"]
**Severity**: SEV1 / SEV2 / SEV3
**Created By**: [Name]

---

## Executive Summary

**Problem**: [What's broken in one sentence]

**Impact**: [Who's affected and how]

**Solution**: [High-level approach]

**ETA**: [Estimated time to resolution]

**Example**:
```
Problem: Database connection pool exhausted due to connection leak
Impact: All users unable to access application (100% downtime)
Solution: Restart application + fix connection leak in code
ETA: 30 minutes (service restored in 5 min, permanent fix in 30 min)
```

---

## Three-Horizon Mitigation

### Immediate (Now - 5 minutes)

**Goal**: Stop the bleeding, restore service immediately

**Actions**:
- [ ] [Action 1]
  - **What**: [Detailed description]
  - **How**: [Commands/steps]
  - **Impact**: [Expected improvement]
  - **Risk**: [Low/Medium/High + explanation]
  - **Rollback**: [How to undo if it fails]
  - **ETA**: [Time to execute]
  - **Owner**: [Who will do this]

**Example**:
```
- [ ] Restart payment service to release connections
  - What: Restart payment service to release database connections
  - How: `systemctl restart payment-service`
  - Impact: All 100 connections released, service restored
  - Risk: Low (stateless service, graceful restart)
  - Rollback: N/A (restart is safe)
  - ETA: 2 minutes
  - Owner: Jane (SRE)

- [ ] Monitor connection pool for 5 minutes
  - What: Verify connections stay below 80%
  - How: `watch -n 5 'psql -c "SELECT count(*) FROM pg_stat_activity"'`
  - Impact: Early detection if issue recurs
  - Risk: None (monitoring only)
  - Rollback: N/A
  - ETA: 5 minutes
  - Owner: Jane (SRE)
```

**Success Criteria**:
- [ ] Service health check passing
- [ ] Users able to access application
- [ ] Connection pool <80% of max
- [ ] No active alerts

---

### Short-term (5 minutes - 1 hour)

**Goal**: Tactical fix to prevent immediate recurrence

**Actions**:
- [ ] [Action 1]
  - **What**: [Detailed description]
  - **How**: [Commands/steps]
  - **Impact**: [Expected improvement]
  - **Risk**: [Low/Medium/High + explanation]
  - **Rollback**: [How to undo if it fails]
  - **ETA**: [Time to execute]
  - **Owner**: [Who will do this]

**Example**:
```
- [ ] Fix connection leak in payment service code
  - What: Add `finally` block to close connection in error path
  - How: Deploy hotfix branch `fix/connection-leak`
  - Impact: Connections properly closed, no leak
  - Risk: Medium (code change requires testing)
  - Rollback: `git revert <commit>` + redeploy
  - ETA: 30 minutes (test + deploy)
  - Owner: Mike (Developer)

- [ ] Increase connection pool size
  - What: Increase max_connections from 100 to 200
  - How: ALTER SYSTEM SET max_connections = 200; SELECT pg_reload_conf();
  - Impact: More headroom for traffic spikes
  - Risk: Low (more connections = more memory, but server has capacity)
  - Rollback: ALTER SYSTEM SET max_connections = 100; SELECT pg_reload_conf();
  - ETA: 5 minutes
  - Owner: Tom (DBA)

- [ ] Add connection pool monitoring alert
  - What: Alert when connections >80% of max
  - How: Create CloudWatch/Grafana alert
  - Impact: Early warning before exhaustion
  - Risk: None (monitoring only)
  - Rollback: Disable alert
  - ETA: 15 minutes
  - Owner: Jane (SRE)
```

**Success Criteria**:
- [ ] Code fix deployed and verified
- [ ] Connection pool increased
- [ ] Monitoring alert configured
- [ ] No recurrence in 1 hour
- [ ] Load test passed (if applicable)

---

### Long-term (1 hour - days/weeks)

**Goal**: Permanent fix and prevention

**Actions**:
- [ ] [Action 1]
  - **What**: [Detailed description]
  - **Priority**: P1 / P2 / P3
  - **Due Date**: [YYYY-MM-DD]
  - **Owner**: [Who will do this]

**Example**:
```
- [ ] Add automated test for connection cleanup
  - What: Integration test that verifies connections are closed in error paths
  - Priority: P1
  - Due Date: 2025-10-27
  - Owner: Lisa (QA)

- [ ] Add connection timeout configuration
  - What: Set connection_timeout = 30s in database config
  - Priority: P2
  - Due Date: 2025-10-28
  - Owner: Tom (DBA)

- [ ] Review all database queries for connection leaks
  - What: Audit all DB queries to ensure proper cleanup
  - Priority: P3
  - Due Date: 2025-11-02
  - Owner: Mike (Developer)

- [ ] Load test for high-traffic events
  - What: Load test with 10x normal traffic to find bottlenecks
  - Priority: P3
  - Due Date: 2025-11-10
  - Owner: John (DevOps)

- [ ] Update runbook with new findings
  - What: Document connection leak troubleshooting steps
  - Priority: P3
  - Due Date: 2025-10-28
  - Owner: Jane (SRE)
```

**Success Criteria**:
- [ ] All P1 actions completed
- [ ] Regression test added (prevents future occurrences)
- [ ] Monitoring improved (detect earlier)
- [ ] Runbook updated
- [ ] Post-mortem published

---

## Risk Assessment

### Risks of Mitigation Actions

| Action | Risk Level | Risk Description | Mitigation |
|--------|------------|------------------|------------|
| [Action 1] | Low/Med/High | [What could go wrong] | [How to reduce risk] |

**Example**:
```
| Restart service | Low | Brief downtime (5s) | Use graceful restart, off-peak time |
| Deploy code fix | Medium | Bug in fix could worsen issue | Test in staging first, have rollback ready |
| Increase connection pool | Low | More memory usage | Server has capacity, monitor memory |
```

### Risks of NOT Mitigating

| Risk | Impact | Probability |
|------|--------|-------------|
| [Risk 1] | [Impact if we do nothing] | High/Med/Low |

**Example**:
```
| Service remains down | All users affected, revenue loss | High (will recur) |
| Connection leak worsens | Database crashes | High |
| SLA breach | Customer refunds, reputation damage | Medium |
```

---

## Communication Plan

### Internal Communication

**Incident Channel**: #incident-YYYYMMDD-title

**Update Frequency**: Every [X] minutes

**Stakeholders to Notify**:
- [ ] Engineering team (#engineering)
- [ ] Customer support (#support)
- [ ] Management (#management)
- [ ] [Other teams]

**Update Template**:
```markdown
[HH:MM] Update:
- Status: [Investigating / Mitigating / Resolved]
- Root Cause: [Known / Under investigation]
- Current Action: [What we're doing now]
- Next Steps: [What's next]
- ETA: [Estimated resolution time]
```

---

### External Communication

**Status Page**: [URL]

**Update Frequency**: Every [X] minutes or when status changes

**Status Page Template**:
```markdown
[HH:MM] Investigating: We are currently investigating [issue description]. Our team is actively working on a resolution.

[HH:MM] Identified: We have identified the issue as [root cause]. We are implementing a fix. ETA: [time].

[HH:MM] Monitoring: The fix has been deployed. We are monitoring to ensure stability.

[HH:MM] Resolved: The issue has been fully resolved. All services are operating normally. We apologize for the inconvenience.
```

**Customer Email** (if needed):
- [ ] Draft email
- [ ] Approve with management
- [ ] Send to affected customers

---

## Validation

### Before Declaring Resolved

Verify all of the following:

- [ ] Root cause identified
- [ ] Immediate fix deployed and verified
- [ ] Service health check passing for >30 minutes
- [ ] Users able to access application
- [ ] Metrics returned to normal (response time, error rate, etc.)
- [ ] No active alerts
- [ ] Load test passed (if applicable)
- [ ] Customer support confirms no ongoing issues

### Monitoring After Resolution

Monitor for [X] hours after declaring resolved:

- [ ] [Metric 1] within normal range
- [ ] [Metric 2] within normal range
- [ ] [Metric 3] within normal range
- [ ] No error spikes
- [ ] No user complaints

**Example**:
```
- [ ] Connection pool <50% of max
- [ ] API response time <200ms (p95)
- [ ] Error rate <0.1%
- [ ] Database CPU <70%
```

---

## Rollback Plan

If mitigation actions fail or make things worse:

### Immediate Rollback

```bash
# Rollback code deployment
git revert <commit>
npm run deploy

# Rollback database config
ALTER SYSTEM SET max_connections = 100;
SELECT pg_reload_conf();

# Verify rollback
curl http://localhost/health
```

### When to Rollback

Rollback if:
- [ ] Issue worsens after mitigation
- [ ] New errors appear
- [ ] Service remains down >X minutes after mitigation
- [ ] Metrics worsen (response time, error rate)

---

## Next Steps

After incident is resolved:

1. [ ] Create post-mortem (within 24 hours)
   - Owner: [Name]
   - Due: [Date]

2. [ ] Schedule post-mortem review meeting
   - Date: [Date]
   - Attendees: [List]

3. [ ] Track action items to completion
   - Use: [JIRA/GitHub/etc.]
   - Review: Weekly in team meeting

4. [ ] Update runbooks based on learnings
   - Owner: [Name]
   - Due: [Date]

5. [ ] Share learnings with organization
   - Format: All-hands presentation / Email / Wiki
   - Owner: [Name]
   - Due: [Date]

---

## Appendix

### Commands Reference

```bash
# Useful commands for this incident
<command1>
<command2>
<command3>
```

### Links

- **Monitoring Dashboard**: [URL]
- **Runbook**: [URL]
- **Related Incidents**: [URL]
- **Incident Channel**: [Slack/Teams URL]

---

**Plan Created**: YYYY-MM-DD HH:MM UTC
**Plan Updated**: YYYY-MM-DD HH:MM UTC
**Status**: Active / Executed / Superseded