243 lines
8.7 KiB
Markdown
243 lines
8.7 KiB
Markdown
---
|
|
name: chaos-engineering-principles
|
|
description: Use when starting chaos engineering, designing fault injection experiments, choosing chaos tools, testing system resilience, or recovering from chaos incidents - provides hypothesis-driven testing, blast radius control, and anti-patterns for safe chaos
|
|
---
|
|
|
|
# Chaos Engineering Principles
|
|
|
|
## Overview
|
|
|
|
**Core principle:** Chaos engineering validates resilience through controlled experiments, not random destruction.
|
|
|
|
**Rule:** Start in staging, with monitoring, with rollback, with small blast radius. No exceptions.
|
|
|
|
## When NOT to Do Chaos
|
|
|
|
Don't run chaos experiments if ANY of these are missing:
|
|
- ❌ No comprehensive monitoring (APM, metrics, logs, alerts)
|
|
- ❌ No automated rollback capability
|
|
- ❌ No baseline metrics documented
|
|
- ❌ No incident response team available
|
|
- ❌ System already unstable (fix stability first)
|
|
- ❌ No staging environment to practice
|
|
|
|
**Fix these prerequisites BEFORE chaos testing.**
|
|
|
|
## Tool Selection Decision Tree
|
|
|
|
| Your Constraint | Choose | Why |
|
|
|----------------|--------|-----|
|
|
| Kubernetes-native, CNCF preference | **LitmusChaos** | Cloud-native, operator-based, excellent K8s integration |
|
|
| Kubernetes-focused, visualization needs | **Chaos Mesh** | Fine-grained control, dashboards, low overhead |
|
|
| Want managed service, quick start | **Gremlin** | Commercial, guided experiments, built-in best practices |
|
|
| Vendor-neutral, maximum flexibility | **Chaos Toolkit** | Open source, plugin ecosystem, any infrastructure |
|
|
| AWS-specific, cost-sensitive | **AWS FIS** | Native AWS integration, pay-per-experiment |
|
|
|
|
**For most teams:** Chaos Toolkit (flexible, free) or Gremlin (fast, managed)
|
|
|
|
## Prerequisites Checklist
|
|
|
|
Before FIRST experiment:
|
|
|
|
**Monitoring (Required):**
|
|
- [ ] Real-time dashboards for key metrics (latency, error rate, throughput)
|
|
- [ ] Distributed tracing for request flows
|
|
- [ ] Log aggregation with timeline correlation
|
|
- [ ] Alerts configured with thresholds
|
|
|
|
**Rollback (Required):**
|
|
- [ ] Automated rollback based on metrics (e.g., error rate > 5% → abort)
|
|
- [ ] Manual kill switch everyone can activate
|
|
- [ ] Rollback tested and documented (< 30 sec recovery)
|
|
|
|
**Baseline (Required):**
|
|
- [ ] Documented normal metrics (P50/P95/P99 latency, error rate %)
|
|
- [ ] Known dependencies and critical paths
|
|
- [ ] System architecture diagram
|
|
|
|
**Team (Required):**
|
|
- [ ] Designated observer monitoring experiment
|
|
- [ ] On-call engineer available
|
|
- [ ] Communication channel established (war room, Slack)
|
|
- [ ] Post-experiment debrief scheduled
|
|
|
|
## Anti-Patterns Catalog
|
|
|
|
### ❌ Production First Chaos
|
|
**Symptom:** "Let's start chaos testing in production to see what breaks"
|
|
|
|
**Why bad:** No practice, no muscle memory, production incidents guaranteed
|
|
|
|
**Fix:** Run 5-10 experiments in staging FIRST. Graduate to production only after proving: experiments work as designed, rollback functions, team can execute response
|
|
|
|
---
|
|
|
|
### ❌ Chaos Without Monitoring
|
|
**Symptom:** "We injected latency but we're not sure what happened"
|
|
|
|
**Why bad:** Blind chaos = no learning. You can't validate resilience without seeing system behavior
|
|
|
|
**Fix:** Set up comprehensive monitoring BEFORE first experiment. Must be able to answer "What changed?" within 30 seconds
|
|
|
|
---
|
|
|
|
### ❌ Unlimited Blast Radius
|
|
**Symptom:** Affecting 100% of traffic/all services on first run
|
|
|
|
**Why bad:** Cascading failures, actual outages, customer impact
|
|
|
|
**Fix:** Start at 0.1-1% traffic. Progression: 0.1% → 1% → 5% → 10% → (stop or 50%). Each step validates before expanding
|
|
|
|
---
|
|
|
|
### ❌ Chaos Without Rollback
|
|
**Symptom:** "The experiment broke everything and we can't stop it"
|
|
|
|
**Why bad:** Chaos becomes real incident, 2+ hour recovery, lost trust
|
|
|
|
**Fix:** Automated abort criteria (error rate threshold, latency threshold, manual kill switch). Test rollback before injecting failures
|
|
|
|
---
|
|
|
|
### ❌ Random Chaos (No Hypothesis)
|
|
**Symptom:** "Let's inject some failures and see what happens"
|
|
|
|
**Why bad:** No learning objective, can't validate resilience, wasted time
|
|
|
|
**Fix:** Every experiment needs hypothesis: "System will [expected behavior] when [failure injected]"
|
|
|
|
## Failure Types Catalog
|
|
|
|
Priority order for microservices:
|
|
|
|
| Failure Type | Priority | Why Test This | Example |
|
|
|--------------|----------|---------------|---------|
|
|
| **Network Latency** | HIGH | Most common production issue | 500ms delay service A → B |
|
|
| **Service Timeout** | HIGH | Tests circuit breakers, retry logic | Service B unresponsive |
|
|
| **Connection Loss** | HIGH | Tests failover, graceful degradation | TCP connection drops |
|
|
| **Resource Exhaustion** | MEDIUM | Tests resource limits, scaling | Memory limit, connection pool full |
|
|
| **Packet Loss** | MEDIUM | Tests retry strategies | 1-10% packet loss |
|
|
| **DNS Failure** | MEDIUM | Tests service discovery resilience | DNS resolution delays |
|
|
| **Cache Failure** | MEDIUM | Tests fallback behavior | Redis down |
|
|
| **Database Errors** | LOW (start) | High risk - test after basics work | Connection refused, query timeout |
|
|
|
|
**Start with network latency** - safest, most informative, easiest rollback.
|
|
|
|
## Experiment Template
|
|
|
|
Use this for every chaos experiment:
|
|
|
|
**1. Hypothesis**
|
|
"If [failure injected], system will [expected behavior], and [metric] will remain [threshold]"
|
|
|
|
Example: "If service-payment experiences 2s latency, circuit breaker will open within 10s, and P99 latency will stay < 500ms"
|
|
|
|
**2. Baseline Metrics**
|
|
- Current P50/P95/P99 latency:
|
|
- Current error rate:
|
|
- Current throughput:
|
|
|
|
**3. Experiment Config**
|
|
- Failure type: [latency / packet loss / service down / etc.]
|
|
- Target: [specific service / % of traffic]
|
|
- Blast radius: [0.1% traffic, single region, canary pods]
|
|
- Duration: [2-5 minutes initial]
|
|
- Abort criteria: [error rate > 5% OR P99 > 1s OR manual stop]
|
|
|
|
**4. Execution**
|
|
- Observer: [name] monitoring dashboards
|
|
- Runner: [name] executing experiment
|
|
- Kill switch: [procedure]
|
|
- Start time: [timestamp]
|
|
|
|
**5. Observation**
|
|
- What happened vs hypothesis:
|
|
- Actual metrics during chaos:
|
|
- System behavior notes:
|
|
|
|
**6. Validation**
|
|
- ✓ Hypothesis validated / ✗ Hypothesis failed
|
|
- Unexpected findings:
|
|
- Action items:
|
|
|
|
## Blast Radius Progression
|
|
|
|
Safe scaling path:
|
|
|
|
| Step | Traffic Affected | Duration | Abort If |
|
|
|------|------------------|----------|----------|
|
|
| **1. Staging** | 100% staging | 5 min | Any production impact |
|
|
| **2. Canary** | 0.1% production | 2 min | Error rate > 1% |
|
|
| **3. Small** | 1% production | 5 min | Error rate > 2% |
|
|
| **4. Medium** | 5% production | 10 min | Error rate > 5% |
|
|
| **5. Large** | 10% production | 15 min | Error rate > 5% |
|
|
|
|
**Never skip steps.** Each step validates before expanding.
|
|
|
|
**Stop at 10-20% for most experiments** - no need to chaos 100% of production traffic.
|
|
|
|
**Low-traffic services (< 1000 req/day):** Use absolute request counts instead of percentages. Minimum 5-10 affected requests per step. Example: 100 req/day service should still start with 5-10 requests (6 hours), not 0.1% (1 request every 10 days).
|
|
|
|
## Your First Experiment (Staging)
|
|
|
|
**Goal:** Build confidence, validate monitoring, test rollback
|
|
|
|
**Experiment:** Network latency on non-critical service
|
|
|
|
```bash
|
|
# Example with Chaos Toolkit
|
|
1. Pick least critical service (e.g., recommendation engine, not payment)
|
|
2. Inject 500ms latency to 100% of staging traffic
|
|
3. Duration: 5 minutes
|
|
4. Expected: Timeouts handled gracefully, fallback behavior activates
|
|
5. Monitor: Error rate, latency, downstream services
|
|
6. Abort if: Error rate > 10% or cascading failures
|
|
7. Debrief: What did we learn? Did monitoring catch it? Did rollback work?
|
|
```
|
|
|
|
**Success criteria:** You can answer "Did our hypothesis hold?" within 5 minutes of experiment completion.
|
|
|
|
## Common Mistakes
|
|
|
|
### ❌ Testing During Incidents
|
|
**Fix:** Only chaos test during stable periods, business hours, with extra staffing
|
|
|
|
---
|
|
|
|
### ❌ Network Latency Underestimation
|
|
**Fix:** Latency cascades - 500ms can become 5s downstream. Start with 100-200ms, observe, then increase
|
|
|
|
---
|
|
|
|
### ❌ No Post-Experiment Review
|
|
**Fix:** Every experiment gets 15-min debrief: What worked? What broke? What did we learn?
|
|
|
|
## Quick Reference
|
|
|
|
**Prerequisites Before First Chaos:**
|
|
1. Monitoring + alerts
|
|
2. Automated rollback
|
|
3. Baseline metrics documented
|
|
4. Team coordinated
|
|
|
|
**Experiment Steps:**
|
|
1. Write hypothesis
|
|
2. Document baseline
|
|
3. Define blast radius (start 0.1%)
|
|
4. Set abort criteria
|
|
5. Execute with observer
|
|
6. Validate hypothesis
|
|
7. Debrief team
|
|
|
|
**Blast Radius Progression:**
|
|
Staging → 0.1% → 1% → 5% → 10% (stop for most experiments)
|
|
|
|
**First Experiment:**
|
|
Network latency (500ms) on non-critical service in staging for 5 minutes
|
|
|
|
## Bottom Line
|
|
|
|
**Chaos engineering is hypothesis-driven science, not random destruction.**
|
|
|
|
Start small (staging, 0.1% traffic), with monitoring, with rollback. Graduate slowly.
|