315 lines
7.0 KiB
Markdown
315 lines
7.0 KiB
Markdown
# Playbook: Disk Full
|
|
|
|
## Symptoms
|
|
|
|
- "No space left on device" errors
|
|
- Applications can't write files
|
|
- Database refuses writes
|
|
- Logs not being written
|
|
- Monitoring alert: "Disk usage >90%"
|
|
|
|
## Severity
|
|
|
|
- **SEV3** if disk >90% but still functioning
|
|
- **SEV2** if disk >95% and applications degraded
|
|
- **SEV1** if disk 100% and applications down
|
|
|
|
## Diagnosis
|
|
|
|
### Step 1: Check Disk Usage
|
|
|
|
```bash
|
|
# Check disk usage by partition
|
|
df -h
|
|
|
|
# Example output:
|
|
# Filesystem Size Used Avail Use% Mounted on
|
|
# /dev/sda1 50G 48G 2G 96% / ← CRITICAL
|
|
# /dev/sdb1 100G 20G 80G 20% /data
|
|
```
|
|
|
|
---
|
|
|
|
### Step 2: Find Large Directories
|
|
|
|
```bash
|
|
# Disk usage by top-level directory
|
|
du -sh /*
|
|
|
|
# Example output:
|
|
# 15G /var ← Likely logs
|
|
# 10G /home
|
|
# 5G /usr
|
|
# 1G /tmp
|
|
|
|
# Drill down into large directory
|
|
du -sh /var/*
|
|
|
|
# Example:
|
|
# 14G /var/log ← FOUND IT
|
|
# 500M /var/cache
|
|
```
|
|
|
|
---
|
|
|
|
### Step 3: Find Large Files
|
|
|
|
```bash
|
|
# Find files larger than 100MB
|
|
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -h -r | head -20
|
|
|
|
# Example output:
|
|
# 5.0G /var/log/application.log ← Large log file
|
|
# 2.0G /var/log/nginx/access.log
|
|
# 500M /tmp/dump.sql
|
|
```
|
|
|
|
---
|
|
|
|
### Step 4: Check for Deleted Files Holding Space
|
|
|
|
```bash
|
|
# Files deleted but process still has handle
|
|
lsof | grep deleted | awk '{print $1, $2, $7}' | sort -u
|
|
|
|
# Example output:
|
|
# nginx 1234 10G ← nginx has handle to 10GB deleted file
|
|
```
|
|
|
|
**Why this happens**:
|
|
- File deleted (`rm /var/log/nginx/access.log`)
|
|
- But process (nginx) still writing to it
|
|
- Disk space not released until process closes file or restarts
|
|
|
|
---
|
|
|
|
## Mitigation
|
|
|
|
### Immediate (Now - 5 min)
|
|
|
|
**Option A: Delete Old Logs**
|
|
```bash
|
|
# Delete old log files (>7 days)
|
|
find /var/log -name "*.log.*" -mtime +7 -delete
|
|
|
|
# Delete compressed logs (>30 days)
|
|
find /var/log -name "*.gz" -mtime +30 -delete
|
|
|
|
# journalctl: Keep only last 7 days
|
|
journalctl --vacuum-time=7d
|
|
|
|
# Impact: Frees disk space immediately
|
|
# Risk: Low (old logs not needed for debugging recent issues)
|
|
```
|
|
|
|
**Option B: Compress Logs**
|
|
```bash
|
|
# Compress large log files
|
|
gzip /var/log/application.log
|
|
gzip /var/log/nginx/access.log
|
|
|
|
# Impact: Reduces log file size by 80-90%
|
|
# Risk: Low (logs still available, just compressed)
|
|
```
|
|
|
|
**Option C: Release Deleted Files**
|
|
```bash
|
|
# Find processes holding deleted files
|
|
lsof | grep deleted
|
|
|
|
# Restart process to release space
|
|
systemctl restart nginx
|
|
|
|
# Or kill and restart
|
|
kill -HUP <PID>
|
|
|
|
# Impact: Frees disk space held by deleted files
|
|
# Risk: Medium (brief service interruption)
|
|
```
|
|
|
|
**Option D: Clean Temp Files**
|
|
```bash
|
|
# Delete old temp files
|
|
rm -rf /tmp/*
|
|
rm -rf /var/tmp/*
|
|
|
|
# Delete apt/yum cache
|
|
apt-get clean # Ubuntu/Debian
|
|
yum clean all # RHEL/CentOS
|
|
|
|
# Delete old kernels (Ubuntu)
|
|
apt-get autoremove --purge
|
|
|
|
# Impact: Frees disk space
|
|
# Risk: Low (temp files can be deleted)
|
|
```
|
|
|
|
---
|
|
|
|
### Short-term (5 min - 1 hour)
|
|
|
|
**Option A: Rotate Logs Immediately**
|
|
```bash
|
|
# Force log rotation
|
|
logrotate -f /etc/logrotate.conf
|
|
|
|
# Verify logs rotated
|
|
ls -lh /var/log/
|
|
|
|
# Configure aggressive rotation (daily instead of weekly)
|
|
# Edit /etc/logrotate.d/application:
|
|
/var/log/application.log {
|
|
daily # Was: weekly
|
|
rotate 7 # Keep 7 days
|
|
compress # Compress old logs
|
|
delaycompress # Don't compress most recent
|
|
missingok # Don't error if file missing
|
|
notifempty # Don't rotate if empty
|
|
create 0640 www-data www-data
|
|
sharedscripts
|
|
postrotate
|
|
systemctl reload application
|
|
endscript
|
|
}
|
|
```
|
|
|
|
**Option B: Archive Old Data**
|
|
```bash
|
|
# Archive old database dumps
|
|
tar -czf old-dumps.tar.gz /backup/*.sql
|
|
rm /backup/*.sql
|
|
|
|
# Move to cheaper storage (S3, Archive)
|
|
aws s3 cp old-dumps.tar.gz s3://archive-bucket/
|
|
rm old-dumps.tar.gz
|
|
|
|
# Impact: Frees local disk space
|
|
# Risk: Low (data archived, not deleted)
|
|
```
|
|
|
|
**Option C: Expand Disk** (cloud)
|
|
```bash
|
|
# AWS: Modify EBS volume
|
|
aws ec2 modify-volume --volume-id vol-1234567890abcdef0 --size 100 # Was 50 GB
|
|
|
|
# Wait for modification to complete (5-10 min)
|
|
watch aws ec2 describe-volumes-modifications --volume-ids vol-1234567890abcdef0
|
|
|
|
# Resize filesystem
|
|
# ext4:
|
|
sudo resize2fs /dev/xvda1
|
|
|
|
# xfs:
|
|
sudo xfs_growfs /
|
|
|
|
# Verify
|
|
df -h
|
|
|
|
# Impact: More disk space
|
|
# Risk: Low (no downtime, but takes time)
|
|
```
|
|
|
|
---
|
|
|
|
### Long-term (1 hour+)
|
|
|
|
- [ ] Add disk usage monitoring (alert at >80%)
|
|
- [ ] Configure log rotation (daily, keep 7 days)
|
|
- [ ] Set up log forwarding (to ELK, Splunk, CloudWatch)
|
|
- [ ] Review disk usage trends (plan capacity)
|
|
- [ ] Add automated cleanup (cron job for old files)
|
|
- [ ] Archive old data (move to S3, Glacier)
|
|
- [ ] Implement log sampling (reduce volume)
|
|
- [ ] Review application logging (reduce verbosity)
|
|
|
|
---
|
|
|
|
## Common Culprits
|
|
|
|
| Location | Cause | Solution |
|
|
|----------|-------|----------|
|
|
| /var/log | Log files not rotated | logrotate, compress, delete old |
|
|
| /tmp | Temp files not cleaned | Delete old files, add cron job |
|
|
| /var/cache | Apt/yum cache | apt-get clean, yum clean all |
|
|
| /home | User files, downloads | Clean up or expand disk |
|
|
| Database | Large tables, no archiving | Archive old data, vacuum |
|
|
| Deleted files | Process holding handle | Restart process |
|
|
|
|
---
|
|
|
|
## Prevention Checklist
|
|
|
|
- [ ] Configure log rotation (daily, 7 days retention)
|
|
- [ ] Add disk monitoring (alert at >80%)
|
|
- [ ] Set up log forwarding (reduce local storage)
|
|
- [ ] Add cron job to clean temp files
|
|
- [ ] Review disk trends monthly
|
|
- [ ] Plan capacity (expand before hitting limit)
|
|
- [ ] Archive old data (move to cheaper storage)
|
|
- [ ] Implement log sampling (reduce volume)
|
|
|
|
---
|
|
|
|
## Escalation
|
|
|
|
**Escalate to developer if**:
|
|
- Application generating excessive logs
|
|
- Need to reduce logging verbosity
|
|
|
|
**Escalate to DBA if**:
|
|
- Database files consuming disk
|
|
- Need to archive old data
|
|
|
|
**Escalate to infrastructure if**:
|
|
- Need to expand disk (physical server)
|
|
- Need to add new disk
|
|
|
|
---
|
|
|
|
## Related Runbooks
|
|
|
|
- [07-service-down.md](07-service-down.md) - If disk full crashed service
|
|
- [../modules/infrastructure.md](../modules/infrastructure.md) - Infrastructure troubleshooting
|
|
|
|
---
|
|
|
|
## Post-Incident
|
|
|
|
After resolving:
|
|
- [ ] Create post-mortem (if SEV1/SEV2)
|
|
- [ ] Identify what filled disk
|
|
- [ ] Implement prevention (log rotation, monitoring)
|
|
- [ ] Review disk trends (prevent recurrence)
|
|
- [ ] Update this runbook if needed
|
|
|
|
---
|
|
|
|
## Useful Commands Reference
|
|
|
|
```bash
|
|
# Disk usage
|
|
df -h # By partition
|
|
du -sh /* # By directory
|
|
du -sh /var/* # Drill down
|
|
|
|
# Large files
|
|
find / -type f -size +100M -exec ls -lh {} \;
|
|
|
|
# Deleted files holding space
|
|
lsof | grep deleted
|
|
|
|
# Clean up
|
|
find /var/log -name "*.log.*" -mtime +7 -delete # Old logs
|
|
gzip /var/log/*.log # Compress
|
|
journalctl --vacuum-time=7d # journalctl
|
|
apt-get clean # Apt cache
|
|
yum clean all # Yum cache
|
|
|
|
# Log rotation
|
|
logrotate -f /etc/logrotate.conf
|
|
|
|
# Expand disk (after EBS resize)
|
|
resize2fs /dev/xvda1 # ext4
|
|
xfs_growfs / # xfs
|
|
```
|