490 lines
10 KiB
Markdown
490 lines
10 KiB
Markdown
# Troubleshooting Runbooks
|
|
|
|
Step-by-step runbooks for resolving common Grey Haven infrastructure issues. Follow procedures systematically for fastest resolution.
|
|
|
|
## Runbook 1: Worker Not Responding
|
|
|
|
### Symptoms
|
|
- API returning 500/502/503 errors
|
|
- Workers timing out or not processing requests
|
|
- Cloudflare error pages showing
|
|
|
|
### Diagnosis Steps
|
|
|
|
**1. Check Cloudflare Status**
|
|
```bash
|
|
# Visit: https://www.cloudflarestatus.com
|
|
# Or query status API
|
|
curl -s https://www.cloudflarestatus.com/api/v2/status.json | jq '.status.indicator'
|
|
```
|
|
|
|
**2. View Worker Logs**
|
|
```bash
|
|
# Real-time logs
|
|
wrangler tail --format pretty
|
|
|
|
# Look for errors:
|
|
# - "Script exceeded CPU time limit"
|
|
# - "Worker threw exception"
|
|
# - "Uncaught TypeError"
|
|
```
|
|
|
|
**3. Check Recent Deployments**
|
|
```bash
|
|
wrangler deployments list
|
|
|
|
# If recent deployment suspicious, rollback:
|
|
wrangler rollback --message "Reverting to stable version"
|
|
```
|
|
|
|
**4. Test Worker Locally**
|
|
```bash
|
|
# Run worker in dev mode
|
|
wrangler dev
|
|
|
|
# Test endpoint
|
|
curl http://localhost:8787/api/health
|
|
```
|
|
|
|
### Resolution Paths
|
|
|
|
**Path A: Platform Issue** - Wait for Cloudflare, monitor status, communicate ETA
|
|
**Path B: Code Error** - Rollback deployment, fix in dev, test before redeploy
|
|
**Path C: Resource Limit** - Check CPU logs, optimize operations, upgrade if needed
|
|
**Path D: Binding Issue** - Verify wrangler.toml, check bindings, redeploy
|
|
|
|
### Prevention
|
|
- Health check endpoint: `GET /health`
|
|
- Monitor error rate with alerts (>1% = alert)
|
|
- Test deployments in staging first
|
|
- Implement circuit breakers for external calls
|
|
|
|
---
|
|
|
|
## Runbook 2: Database Connection Failures
|
|
|
|
### Symptoms
|
|
- "connection refused" errors
|
|
- "too many connections" errors
|
|
- Application timing out on database queries
|
|
- 503 errors from API
|
|
|
|
### Diagnosis Steps
|
|
|
|
**1. Test Database Connection**
|
|
```bash
|
|
# Direct connection test
|
|
pscale shell greyhaven-db main
|
|
|
|
# If fails, check:
|
|
# - Database status
|
|
# - Credentials
|
|
# - Network connectivity
|
|
```
|
|
|
|
**2. Check Connection Pool**
|
|
```bash
|
|
# Query pool status
|
|
curl http://localhost:8000/pool-status
|
|
|
|
# Expected healthy response:
|
|
{
|
|
"size": 50,
|
|
"checked_out": 25, # <80% is healthy
|
|
"overflow": 0,
|
|
"available": 25
|
|
}
|
|
```
|
|
|
|
**3. Check Active Connections**
|
|
```sql
|
|
-- In pscale shell
|
|
SELECT
|
|
COUNT(*) as active,
|
|
MAX(query_start) as oldest_query
|
|
FROM pg_stat_activity
|
|
WHERE state = 'active';
|
|
|
|
-- If active = pool size, pool exhausted
|
|
-- If oldest_query >10min, leaked connection
|
|
```
|
|
|
|
**4. Review Application Logs**
|
|
```bash
|
|
# Search for connection errors
|
|
grep -i "connection" logs/app.log | tail -50
|
|
|
|
# Common errors:
|
|
# - "Pool timeout"
|
|
# - "Connection refused"
|
|
# - "Max connections reached"
|
|
```
|
|
|
|
### Resolution Paths
|
|
|
|
**Path A: Invalid Credentials**
|
|
```bash
|
|
# Rotate credentials
|
|
pscale password create greyhaven-db main app-password
|
|
|
|
# Update environment variable
|
|
# Restart application
|
|
```
|
|
|
|
**Path B: Pool Exhausted**
|
|
```python
|
|
# Increase pool size in database.py
|
|
engine = create_engine(
|
|
database_url,
|
|
pool_size=50, # Increase from 20
|
|
max_overflow=20
|
|
)
|
|
```
|
|
|
|
**Path C: Connection Leaks**
|
|
```python
|
|
# Fix: Use context managers
|
|
with Session(engine) as session:
|
|
# Work with session
|
|
pass # Automatically closed
|
|
```
|
|
|
|
**Path D: Database Paused/Down**
|
|
```bash
|
|
# Resume database if paused
|
|
pscale database resume greyhaven-db
|
|
|
|
# Check database status
|
|
pscale database show greyhaven-db
|
|
```
|
|
|
|
### Prevention
|
|
- Use connection pooling with proper limits
|
|
- Implement retry logic with exponential backoff
|
|
- Monitor pool utilization (alert >80%)
|
|
- Test for connection leaks in CI/CD
|
|
|
|
---
|
|
|
|
## Runbook 3: Deployment Failures
|
|
|
|
### Symptoms
|
|
- `wrangler deploy` fails
|
|
- CI/CD pipeline fails at deployment step
|
|
- New code not reflecting in production
|
|
|
|
### Diagnosis Steps
|
|
|
|
**1. Check Deployment Error**
|
|
```bash
|
|
wrangler deploy --verbose
|
|
|
|
# Common errors:
|
|
# - "Script exceeds size limit"
|
|
# - "Syntax error in worker"
|
|
# - "Environment variable missing"
|
|
# - "Binding not found"
|
|
```
|
|
|
|
**2. Verify Build Output**
|
|
```bash
|
|
# Check built file
|
|
ls -lh dist/
|
|
npm run build
|
|
|
|
# Ensure build succeeds locally
|
|
```
|
|
|
|
**3. Check Environment Variables**
|
|
```bash
|
|
# List secrets
|
|
wrangler secret list
|
|
|
|
# Verify wrangler.toml vars
|
|
cat wrangler.toml | grep -A 10 "\[vars\]"
|
|
```
|
|
|
|
**4. Test Locally**
|
|
```bash
|
|
# Start dev server
|
|
wrangler dev
|
|
|
|
# If works locally but not production:
|
|
# - Environment variable mismatch
|
|
# - Binding configuration issue
|
|
```
|
|
|
|
### Resolution Paths
|
|
|
|
**Path A: Bundle Too Large**
|
|
```bash
|
|
# Check bundle size
|
|
ls -lh dist/worker.js
|
|
|
|
# Solutions:
|
|
# - Tree shake unused code
|
|
# - Code split large modules
|
|
# - Use fetch instead of SDK
|
|
```
|
|
|
|
**Path B: Syntax Error**
|
|
```bash
|
|
# Run TypeScript check
|
|
npm run type-check
|
|
|
|
# Run linter
|
|
npm run lint
|
|
|
|
# Fix errors before deploying
|
|
```
|
|
|
|
**Path C: Missing Variables**
|
|
```bash
|
|
# Add missing secret
|
|
wrangler secret put API_KEY
|
|
|
|
# Or add to wrangler.toml vars
|
|
[vars]
|
|
API_ENDPOINT = "https://api.example.com"
|
|
```
|
|
|
|
**Path D: Binding Not Found**
|
|
```toml
|
|
# wrangler.toml - Add binding
|
|
[[kv_namespaces]]
|
|
binding = "CACHE"
|
|
id = "abc123"
|
|
|
|
[[d1_databases]]
|
|
binding = "DB"
|
|
database_name = "greyhaven-db"
|
|
database_id = "xyz789"
|
|
```
|
|
|
|
### Prevention
|
|
- Bundle size check in CI/CD
|
|
- Pre-commit hooks for validation
|
|
- Staging environment for testing
|
|
- Automated deployment tests
|
|
|
|
---
|
|
|
|
## Runbook 4: Performance Degradation
|
|
|
|
### Symptoms
|
|
- API response times increased (>2x normal)
|
|
- Slow page loads
|
|
- User complaints about slowness
|
|
- Timeout errors
|
|
|
|
### Diagnosis Steps
|
|
|
|
**1. Check Current Latency**
|
|
```bash
|
|
# Test endpoint
|
|
curl -w "\nTotal: %{time_total}s\n" -o /dev/null -s https://api.greyhaven.io/orders
|
|
|
|
# p95 should be <500ms
|
|
# If >1s, investigate
|
|
```
|
|
|
|
**2. Analyze Worker Logs**
|
|
```bash
|
|
wrangler tail --format json | jq '{duration: .outcome.duration, event: .event}'
|
|
|
|
# Identify slow requests
|
|
# Check what's taking time
|
|
```
|
|
|
|
**3. Check Database Queries**
|
|
```bash
|
|
# Slow query log
|
|
pscale database insights greyhaven-db main --slow-queries
|
|
|
|
# Look for:
|
|
# - N+1 queries (many small queries)
|
|
# - Missing indexes (full table scans)
|
|
# - Long-running queries (>100ms)
|
|
```
|
|
|
|
**4. Profile Application**
|
|
```bash
|
|
# Add timing middleware
|
|
# Log slow operations
|
|
# Identify bottleneck (DB, API, compute)
|
|
```
|
|
|
|
### Resolution Paths
|
|
|
|
**Path A: N+1 Queries**
|
|
```python
|
|
# Use eager loading
|
|
statement = (
|
|
select(Order)
|
|
.options(selectinload(Order.items))
|
|
)
|
|
```
|
|
|
|
**Path B: Missing Indexes**
|
|
```sql
|
|
-- Add indexes
|
|
CREATE INDEX idx_orders_user_id ON orders(user_id);
|
|
CREATE INDEX idx_items_order_id ON order_items(order_id);
|
|
```
|
|
|
|
**Path C: No Caching**
|
|
```typescript
|
|
// Add Redis caching
|
|
const cached = await redis.get(cacheKey);
|
|
if (cached) return cached;
|
|
|
|
const result = await expensiveOperation();
|
|
await redis.setex(cacheKey, 300, result);
|
|
```
|
|
|
|
**Path D: Worker CPU Limit**
|
|
```typescript
|
|
// Optimize expensive operations
|
|
// Use async operations
|
|
// Offload to external service
|
|
```
|
|
|
|
### Prevention
|
|
- Monitor p95 latency (alert >500ms)
|
|
- Test for N+1 queries in CI/CD
|
|
- Add indexes for foreign keys
|
|
- Implement caching layer
|
|
- Performance budgets in tests
|
|
|
|
---
|
|
|
|
## Runbook 5: Network Connectivity Issues
|
|
|
|
### Symptoms
|
|
- Intermittent failures
|
|
- DNS resolution errors
|
|
- Connection timeouts
|
|
- CORS errors
|
|
|
|
### Diagnosis Steps
|
|
|
|
**1. Test DNS Resolution**
|
|
```bash
|
|
# Check DNS
|
|
nslookup api.partner.com
|
|
dig api.partner.com
|
|
|
|
# Measure DNS time
|
|
time nslookup api.partner.com
|
|
|
|
# If >1s, DNS is slow
|
|
```
|
|
|
|
**2. Test Connectivity**
|
|
```bash
|
|
# Basic connectivity
|
|
ping api.partner.com
|
|
|
|
# Trace route
|
|
traceroute api.partner.com
|
|
|
|
# Full timing breakdown
|
|
curl -w "\nDNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTotal: %{time_total}s\n" \
|
|
-o /dev/null -s https://api.partner.com
|
|
```
|
|
|
|
**3. Check CORS**
|
|
```bash
|
|
# Preflight request
|
|
curl -I -X OPTIONS https://api.greyhaven.io/api/users \
|
|
-H "Origin: https://app.greyhaven.io" \
|
|
-H "Access-Control-Request-Method: POST"
|
|
|
|
# Verify headers:
|
|
# - Access-Control-Allow-Origin
|
|
# - Access-Control-Allow-Methods
|
|
```
|
|
|
|
**4. Check Firewall/Security**
|
|
```bash
|
|
# Test from different location
|
|
# Check IP whitelist
|
|
# Verify SSL certificate
|
|
```
|
|
|
|
### Resolution Paths
|
|
|
|
**Path A: Slow DNS**
|
|
```typescript
|
|
// Implement DNS caching
|
|
const DNS_CACHE = new Map();
|
|
// Cache DNS for 60s
|
|
```
|
|
|
|
**Path B: Connection Timeout**
|
|
```typescript
|
|
// Increase timeout
|
|
const controller = new AbortController();
|
|
setTimeout(() => controller.abort(), 30000); // 30s
|
|
```
|
|
|
|
**Path C: CORS Error**
|
|
```typescript
|
|
// Add CORS headers
|
|
response.headers.set('Access-Control-Allow-Origin', origin);
|
|
response.headers.set('Access-Control-Allow-Methods', 'GET,POST,PUT,DELETE');
|
|
```
|
|
|
|
**Path D: SSL/TLS Issue**
|
|
```bash
|
|
# Check certificate
|
|
openssl s_client -connect api.partner.com:443
|
|
|
|
# Verify not expired
|
|
# Check certificate chain
|
|
```
|
|
|
|
### Prevention
|
|
- DNS caching (60s TTL)
|
|
- Appropriate timeouts (30s for external APIs)
|
|
- Health checks for external dependencies
|
|
- Circuit breakers for failures
|
|
- Monitor external API latency
|
|
|
|
---
|
|
|
|
## Emergency Procedures (SEV1)
|
|
|
|
**Immediate Actions**:
|
|
1. **Assess**: Users affected? Functionality broken? Data loss risk?
|
|
2. **Communicate**: Alert team, update status page
|
|
3. **Stop Bleeding**: `wrangler rollback` or disable feature
|
|
4. **Diagnose**: Logs, recent changes, metrics
|
|
5. **Fix**: Hotfix or workaround, test first
|
|
6. **Verify**: Monitor metrics, test functionality
|
|
7. **Postmortem**: Document, root cause, prevention
|
|
|
|
---
|
|
|
|
## Escalation Matrix
|
|
|
|
| Issue Type | First Response | Escalate To | Escalation Trigger |
|
|
|------------|---------------|-------------|-------------------|
|
|
| Worker errors | DevOps troubleshooter | incident-responder | SEV1/SEV2 |
|
|
| Performance | DevOps troubleshooter | performance-optimizer | >30min unresolved |
|
|
| Database | DevOps troubleshooter | data-validator | Schema issues |
|
|
| Security | DevOps troubleshooter | security-analyzer | Breach suspected |
|
|
| Application bugs | DevOps troubleshooter | smart-debug | Infrastructure ruled out |
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- **Examples**: [Examples Index](../examples/INDEX.md) - Full troubleshooting examples
|
|
- **Diagnostic Commands**: [diagnostic-commands.md](diagnostic-commands.md) - Command reference
|
|
- **Cloudflare Guide**: [cloudflare-workers-guide.md](cloudflare-workers-guide.md) - Platform-specific
|
|
|
|
---
|
|
|
|
Return to [reference index](INDEX.md)
|