294 lines
7.0 KiB
Markdown
294 lines
7.0 KiB
Markdown
# Playbook: DDoS Attack
|
|
|
|
## Symptoms
|
|
|
|
- Sudden traffic spike (10x-100x normal)
|
|
- Legitimate users can't access service
|
|
- High bandwidth usage (saturated)
|
|
- Server overload (CPU, memory, network)
|
|
- Monitoring alert: "Traffic spike", "Bandwidth >90%"
|
|
|
|
## Severity
|
|
|
|
- **SEV1** - Production service unavailable due to attack
|
|
|
|
## Diagnosis
|
|
|
|
### Step 1: Confirm Traffic Spike
|
|
|
|
```bash
|
|
# Check current connections
|
|
netstat -ntu | wc -l
|
|
|
|
# Compare to baseline (normal: 100-500, attack: 10,000+)
|
|
|
|
# Check requests per second (nginx)
|
|
tail -f /var/log/nginx/access.log | awk '{print $4}' | uniq -c
|
|
```
|
|
|
|
---
|
|
|
|
### Step 2: Identify Attack Pattern
|
|
|
|
**Check connections by IP**:
|
|
```bash
|
|
# Top 20 IPs by connection count
|
|
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20
|
|
|
|
# Example output:
|
|
# 5000 192.168.1.100 ← Attacker IP
|
|
# 3000 192.168.1.101 ← Attacker IP
|
|
# 2 192.168.1.200 ← Legitimate user
|
|
```
|
|
|
|
**Check HTTP requests by IP** (nginx):
|
|
```bash
|
|
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -20
|
|
```
|
|
|
|
**Check request patterns**:
|
|
```bash
|
|
# Check requested URLs
|
|
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -20
|
|
|
|
# Check user agents (bots often have telltale user agents)
|
|
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -nr
|
|
```
|
|
|
|
---
|
|
|
|
### Step 3: Classify Attack Type
|
|
|
|
**HTTP Flood** (application layer):
|
|
- Many HTTP requests from distributed IPs
|
|
- Valid HTTP requests, just too many
|
|
- Example: 10,000 requests/second to homepage
|
|
|
|
**SYN Flood** (network layer):
|
|
- Many TCP SYN packets
|
|
- Connection requests never complete
|
|
- Exhausts server connection table
|
|
|
|
**Amplification** (DNS, NTP):
|
|
- Small request → Large response
|
|
- Attacker spoofs your IP
|
|
- Servers send large responses to you
|
|
|
|
---
|
|
|
|
## Mitigation
|
|
|
|
### Immediate (Now - 5 min)
|
|
|
|
**Option A: Block Attacker IPs** (if few IPs)
|
|
```bash
|
|
# Block single IP (iptables)
|
|
iptables -A INPUT -s <ATTACKER_IP> -j DROP
|
|
|
|
# Block IP range
|
|
iptables -A INPUT -s 192.168.1.0/24 -j DROP
|
|
|
|
# Block specific country (using ipset + GeoIP)
|
|
# Advanced, see infrastructure team
|
|
|
|
# Impact: Blocks attacker, restores service
|
|
# Risk: Low (if attacker IPs identified correctly)
|
|
```
|
|
|
|
**Option B: Enable Rate Limiting** (nginx)
|
|
```nginx
|
|
# Add to nginx.conf
|
|
http {
|
|
# Define rate limit zone (10 req/s per IP)
|
|
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
|
|
|
|
server {
|
|
location / {
|
|
# Apply rate limit
|
|
limit_req zone=one burst=20 nodelay;
|
|
limit_req_status 429;
|
|
}
|
|
}
|
|
}
|
|
|
|
# Reload nginx
|
|
nginx -t && systemctl reload nginx
|
|
|
|
# Impact: Limits requests per IP
|
|
# Risk: Low (legitimate users rarely exceed 10 req/s)
|
|
```
|
|
|
|
**Option C: Enable CloudFlare "Under Attack" Mode**
|
|
```bash
|
|
# If using CloudFlare:
|
|
# 1. Log in to CloudFlare dashboard
|
|
# 2. Select domain
|
|
# 3. Click "Under Attack Mode"
|
|
# 4. Adds JavaScript challenge before serving content
|
|
|
|
# Impact: Blocks bots, allows legitimate browsers
|
|
# Risk: Low (slight user friction)
|
|
```
|
|
|
|
**Option D: Enable AWS Shield** (AWS)
|
|
```bash
|
|
# AWS Shield Standard: Free, automatic DDoS protection
|
|
# AWS Shield Advanced: $3000/month, enhanced protection
|
|
|
|
# CloudFormation:
|
|
aws cloudformation deploy \
|
|
--template-file shield.yaml \
|
|
--stack-name ddos-protection
|
|
|
|
# Impact: Absorbs DDoS at AWS edge
|
|
# Risk: None (AWS handles)
|
|
```
|
|
|
|
---
|
|
|
|
### Short-term (5 min - 1 hour)
|
|
|
|
**Option A: Add Connection Limits**
|
|
```nginx
|
|
# Limit concurrent connections per IP
|
|
limit_conn_zone $binary_remote_addr zone=addr:10m;
|
|
|
|
server {
|
|
location / {
|
|
limit_conn addr 10; # Max 10 concurrent connections per IP
|
|
}
|
|
}
|
|
```
|
|
|
|
**Option B: Add CAPTCHA** (reCAPTCHA)
|
|
```html
|
|
<!-- Add reCAPTCHA to sensitive endpoints -->
|
|
<form action="/login" method="POST">
|
|
<input type="email" name="email">
|
|
<input type="password" name="password">
|
|
<div class="g-recaptcha" data-sitekey="YOUR_SITE_KEY"></div>
|
|
<button type="submit">Login</button>
|
|
</form>
|
|
```
|
|
|
|
**Option C: Scale Up** (cloud auto-scaling)
|
|
```bash
|
|
# AWS: Increase Auto Scaling Group desired capacity
|
|
aws autoscaling set-desired-capacity \
|
|
--auto-scaling-group-name my-asg \
|
|
--desired-capacity 20 # Was 5
|
|
|
|
# Impact: More capacity to handle attack
|
|
# Risk: Medium (costs money, may not fully mitigate)
|
|
# NOTE: Only do this if legitimate traffic also spiked
|
|
```
|
|
|
|
---
|
|
|
|
### Long-term (1 hour+)
|
|
|
|
- [ ] Enable CloudFlare or AWS Shield (DDoS protection service)
|
|
- [ ] Implement rate limiting on all endpoints
|
|
- [ ] Add CAPTCHA to login, signup, checkout
|
|
- [ ] Configure auto-scaling (handle legitimate traffic spikes)
|
|
- [ ] Add monitoring alert for traffic anomalies
|
|
- [ ] Create DDoS response plan
|
|
- [ ] Contact ISP for upstream filtering (if very large attack)
|
|
- [ ] Review and update firewall rules
|
|
- [ ] Add geographic blocking (if applicable)
|
|
|
|
---
|
|
|
|
## Important Notes
|
|
|
|
**DO NOT**:
|
|
- Scale up indefinitely (attack can grow, costs explode)
|
|
- Fight DDoS at application layer alone (use CDN, cloud protection)
|
|
|
|
**DO**:
|
|
- Use CDN/DDoS protection service (CloudFlare, AWS Shield, Akamai)
|
|
- Enable rate limiting
|
|
- Block attacker IPs/ranges
|
|
- Monitor costs (auto-scaling can be expensive)
|
|
|
|
---
|
|
|
|
## Escalation
|
|
|
|
**Escalate to infrastructure team if**:
|
|
- Attack very large (>10 Gbps)
|
|
- Need upstream filtering at ISP level
|
|
|
|
**Escalate to security team**:
|
|
- All DDoS attacks (for post-mortem, legal action)
|
|
|
|
**Contact ISP if**:
|
|
- Attack saturating internet connection
|
|
- Need transit provider to filter
|
|
|
|
**Contact CloudFlare/AWS if**:
|
|
- Using their DDoS protection
|
|
- Need assistance enabling features
|
|
|
|
---
|
|
|
|
## Prevention Checklist
|
|
|
|
- [ ] Use CDN (CloudFlare, CloudFront, Akamai)
|
|
- [ ] Enable DDoS protection (AWS Shield, CloudFlare)
|
|
- [ ] Implement rate limiting (per IP, per user)
|
|
- [ ] Add CAPTCHA to sensitive endpoints
|
|
- [ ] Configure auto-scaling (within cost limits)
|
|
- [ ] Monitor traffic patterns (detect spikes early)
|
|
- [ ] Have DDoS response plan ready
|
|
- [ ] Test response plan (tabletop exercise)
|
|
|
|
---
|
|
|
|
## Related Runbooks
|
|
|
|
- [01-high-cpu-usage.md](01-high-cpu-usage.md) - If CPU overloaded
|
|
- [07-service-down.md](07-service-down.md) - If service crashed
|
|
- [../modules/security-incidents.md](../modules/security-incidents.md) - Security response
|
|
- [../modules/infrastructure.md](../modules/infrastructure.md) - Infrastructure troubleshooting
|
|
|
|
---
|
|
|
|
## Post-Incident
|
|
|
|
After resolving:
|
|
- [ ] Create post-mortem (mandatory for DDoS)
|
|
- [ ] Identify attack vectors
|
|
- [ ] Document attacker IPs, patterns
|
|
- [ ] Report to ISP, CloudFlare (they may block attacker)
|
|
- [ ] Review and improve DDoS defenses
|
|
- [ ] Consider legal action (if attacker identified)
|
|
- [ ] Update this runbook if needed
|
|
|
|
---
|
|
|
|
## Useful Commands Reference
|
|
|
|
```bash
|
|
# Check connection count
|
|
netstat -ntu | wc -l
|
|
|
|
# Top IPs by connection count
|
|
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr | head -20
|
|
|
|
# Block IP (iptables)
|
|
iptables -A INPUT -s <IP> -j DROP
|
|
|
|
# Check nginx requests per second
|
|
tail -f /var/log/nginx/access.log | awk '{print $4}' | uniq -c
|
|
|
|
# List iptables rules
|
|
iptables -L -n -v
|
|
|
|
# Clear all iptables rules (CAREFUL!)
|
|
iptables -F
|
|
|
|
# Save iptables rules (persist after reboot)
|
|
iptables-save > /etc/iptables/rules.v4
|
|
```
|