gh-anton-abyzov-specweave-plugins-specweave-infrastructure/agents/sre/playbooks/03-memory-leak.md

# Playbook: Memory Leak

## Symptoms

- Memory usage increasing continuously over time
- Application crashes with OutOfMemoryError (Java) or "JavaScript heap out of memory" (Node.js)
- Performance degrades over time
- High swap usage
- Monitoring alert: "Memory usage >90%"

## Severity

- **SEV2** if memory increasing but not yet critical
- **SEV1** if application crashed or unresponsive

## Diagnosis

### Step 1: Confirm Memory Leak

```bash
# Monitor memory over time (5 minute intervals)
watch -n 300 'ps aux | grep <process> | awk "{print \$4, \$5, \$6}"'

# Check if memory continuously increasing
# Leak: 20% → 30% → 40% → 50% (linear growth)
# Normal: 30% → 32% → 31% → 30% (stable)
```

---

### Step 2: Get Memory Snapshot

**Java (Heap Dump)**:
```bash
# Get heap dump
jmap -dump:format=b,file=heap.bin <PID>

# Analyze with jhat or VisualVM
jhat heap.bin
# Open http://localhost:7000

# Or use Eclipse Memory Analyzer
```

**Node.js (Heap Snapshot)**:
```bash
# Start with --inspect
node --inspect index.js

# Chrome DevTools → Memory → Take heap snapshot

# Or use heapdump module
const heapdump = require('heapdump');
heapdump.writeSnapshot('/tmp/heap-' + Date.now() + '.heapsnapshot');
```

**Python (Memory Profiler)**:
```bash
# Install memory_profiler
pip install memory_profiler

# Profile function
python -m memory_profiler script.py
```

---

### Step 3: Identify Leak Source

**Look for**:
- Large arrays/objects growing over time
- Detached DOM nodes (if browser/UI)
- Event listeners not removed
- Timers/intervals not cleared
- Closures holding references
- Cache without eviction policy

**Common patterns**:
```javascript
// 1. Global cache growing forever
global.cache = {}; // Never cleared

// 2. Event listeners not removed
emitter.on('event', handler); // Never removed

// 3. Timers not cleared
setInterval(() => { /* ... */ }, 1000); // Never cleared

// 4. Closures
function createHandler() {
  const largeData = new Array(1000000);
  return () => {
    // Closure keeps largeData in memory
  };
}
```

---

## Mitigation

### Immediate (Now - 5 min)

**Option A: Restart Application**
```bash
# Restart to free memory
systemctl restart application

# Impact: Memory usage returns to baseline
# Risk: Low (brief downtime)
# NOTE: This is temporary, leak will recur!
```

**Option B: Increase Memory Limit** (temporary)
```bash
# Java
java -Xmx4G -jar application.jar  # Was 2G

# Node.js
node --max-old-space-size=4096 index.js  # Was 2048

# Impact: Buys time to find root cause
# Risk: Low (but doesn't fix leak)
```

**Option C: Scale Horizontally** (cloud)
```bash
# Add more instances
# Use load balancer to rotate traffic
# Restart instances on schedule (e.g., every 6 hours)

# Impact: Distributes load, restarts prevent OOM
# Risk: Low (but doesn't fix root cause)
```

---

### Short-term (5 min - 1 hour)

**Analyze heap dump** and identify leak source

**Common Fixes**:

**1. Add LRU Cache**
```javascript
// BAD: Unbounded cache
const cache = {};

// GOOD: LRU cache with size limit
const LRU = require('lru-cache');
const cache = new LRU({ max: 1000 });
```

**2. Remove Event Listeners**
```javascript
// Add listener
const handler = () => { /* ... */ };
emitter.on('event', handler);

// CRITICAL: Remove later
emitter.off('event', handler);

// React/Vue: cleanup in componentWillUnmount/onUnmounted
```

**3. Clear Timers**
```javascript
// Set timer
const intervalId = setInterval(() => { /* ... */ }, 1000);

// CRITICAL: Clear later
clearInterval(intervalId);

// React: cleanup in useEffect return
useEffect(() => {
  const id = setInterval(() => { /* ... */ }, 1000);
  return () => clearInterval(id);
}, []);
```

**4. Close Connections**
```javascript
// BAD: Connection leak
const conn = await db.connect();
await conn.query(/* ... */);
// Connection never closed!

// GOOD: Always close
const conn = await db.connect();
try {
  await conn.query(/* ... */);
} finally {
  await conn.close(); // CRITICAL
}
```

---

### Long-term (1 hour+)

- [ ] Add memory monitoring (alert if >80% and increasing)
- [ ] Add memory profiling to CI/CD (detect leaks early)
- [ ] Use WeakMap for caches (auto garbage collected)
- [ ] Review closure usage (avoid holding large data)
- [ ] Add automated restart (every N hours, if leak can't be fixed immediately)
- [ ] Load test to reproduce leak in test environment
- [ ] Fix root cause in code

---

## Escalation

**Escalate to developer if**:
- Application code causing leak
- Requires code fix

**Escalate to platform team if**:
- Platform/framework bug
- Requires upgrade or workaround

---

## Prevention Checklist

- [ ] Use LRU cache (not unbounded)
- [ ] Remove event listeners in cleanup
- [ ] Clear timers/intervals
- [ ] Close database connections (use `finally`)
- [ ] Avoid closures holding large data
- [ ] Use WeakMap for temporary caches
- [ ] Profile memory in development
- [ ] Load test before production

---

## Related Runbooks

- [01-high-cpu-usage.md](01-high-cpu-usage.md) - If CPU also high
- [07-service-down.md](07-service-down.md) - If OOM crashed service
- [../modules/backend-diagnostics.md](../modules/backend-diagnostics.md) - Backend troubleshooting

---

## Post-Incident

After resolving:
- [ ] Create post-mortem
- [ ] Identify leak source from heap dump
- [ ] Fix code
- [ ] Add regression test (memory profiling)
- [ ] Add monitoring alert
- [ ] Update this runbook if needed