gh-ahmedasmar-devops-claude-skills-monitoring-observability/references/slo_sla_guide.md

SLI, SLO, and SLA Guide

Definitions

SLI (Service Level Indicator)

What: A quantitative measure of service quality

Examples:

• Request latency (ms)
• Error rate (%)
• Availability (%)
• Throughput (requests/sec)

SLO (Service Level Objective)

What: Target value or range for an SLI

Examples:

• "99.9% of requests return in < 500ms"
• "99.95% availability"
• "Error rate < 0.1%"

SLA (Service Level Agreement)

What: Business contract with consequences for SLO violations

Examples:

• "99.9% uptime or 10% monthly credit"
• "p95 latency < 1s or refund"

Relationship

SLI = Measurement
SLO = Target (internal goal)
SLA = Promise (customer contract with penalties)

Example:
SLI: Actual availability this month = 99.92%
SLO: Target availability = 99.9%
SLA: Guaranteed availability = 99.5% (with penalties)

---

Choosing SLIs

The Four Golden Signals as SLIs

1. Latency SLIs

   • Request duration (p50, p95, p99)
   • Time to first byte
   • Page load time

2. Availability/Success SLIs

   • % of successful requests
   • % uptime
   • % of requests completing

3. Throughput SLIs (less common)

   • Requests per second
   • Transactions per second

4. Saturation SLIs (internal only)

   • Resource utilization
   • Queue depth

SLI Selection Criteria

✅ Good SLIs:

• Measured from user perspective
• Directly impact user experience
• Aggregatable across instances
• Proportional to user happiness

❌ Bad SLIs:

• Internal metrics only
• Not user-facing
• Hard to measure consistently

Examples by Service Type

Web Application:

SLI 1: Request Success Rate
  = successful_requests / total_requests

SLI 2: Request Latency (p95)
  = 95th percentile of response times

SLI 3: Availability
  = time_service_responding / total_time

API Service:

SLI 1: Error Rate
  = (4xx_errors + 5xx_errors) / total_requests

SLI 2: Response Time (p99)
  = 99th percentile latency

SLI 3: Throughput
  = requests_per_second

Batch Processing:

SLI 1: Job Success Rate
  = successful_jobs / total_jobs

SLI 2: Processing Latency
  = time_from_submission_to_completion

SLI 3: Freshness
  = age_of_oldest_unprocessed_item

Storage Service:

SLI 1: Durability
  = data_not_lost / total_data

SLI 2: Read Latency (p99)
  = 99th percentile read time

SLI 3: Write Success Rate
  = successful_writes / total_writes

---

Setting SLO Targets

Start with Current Performance

1. Measure baseline: Collect 30 days of data
2. Analyze distribution: Look at p50, p95, p99, p99.9
3. Set initial SLO: Slightly better than worst performer
4. Iterate: Tighten or loosen based on feasibility

Example Process

Current Performance (30 days):

p50 latency: 120ms
p95 latency: 450ms
p99 latency: 1200ms
p99.9 latency: 3500ms

Error rate: 0.05%
Availability: 99.95%

Initial SLOs:

Latency: p95 < 500ms (slightly worse than current p95)
Error rate: < 0.1% (double current rate)
Availability: 99.9% (slightly worse than current)

Rationale: Start loose, prevent false alarms, tighten over time

Common SLO Targets

Availability:

• 99% (3.65 days downtime/year): Internal tools
• 99.5% (1.83 days/year): Non-critical services
• 99.9% (8.76 hours/year): Standard production
• 99.95% (4.38 hours/year): Critical services
• 99.99% (52 minutes/year): High availability
• 99.999% (5 minutes/year): Mission critical

Latency:

• p50 < 100ms: Excellent responsiveness
• p95 < 500ms: Standard web applications
• p99 < 1s: Acceptable for most users
• p99.9 < 5s: Acceptable for rare edge cases

Error Rate:

• < 0.01% (99.99% success): Critical operations
• < 0.1% (99.9% success): Standard production
• < 1% (99% success): Non-critical services

---

Error Budgets

Concept

Error budget = (100% - SLO target)

If SLO is 99.9%, error budget is 0.1%

Purpose: Balance reliability with feature velocity

Calculation

For availability:

Monthly error budget = (1 - SLO) × time_period

Example (99.9% SLO, 30 days):
Error budget = 0.001 × 30 days = 0.03 days = 43.2 minutes

For request-based SLIs:

Error budget = (1 - SLO) × total_requests

Example (99.9% SLO, 10M requests/month):
Error budget = 0.001 × 10,000,000 = 10,000 failed requests

Error Budget Consumption

Formula:

Budget consumed = actual_errors / allowed_errors × 100%

Example:
SLO: 99.9% (0.1% error budget)
Total requests: 1,000,000
Failed requests: 500
Allowed failures: 1,000

Budget consumed = 500 / 1,000 × 100% = 50%
Budget remaining = 50%

Error Budget Policy

Example policy:

## Error Budget Policy

### If error budget > 50%
- Deploy frequently (multiple times per day)
- Take calculated risks
- Experiment with new features
- Acceptable to have some incidents

### If error budget 20-50%
- Deploy normally (once per day)
- Increase testing
- Review recent changes
- Monitor closely

### If error budget < 20%
- Freeze non-critical deploys
- Focus on reliability improvements
- Postmortem all incidents
- Reduce change velocity

### If error budget exhausted (< 0%)
- Complete deploy freeze except rollbacks
- All hands on reliability
- Mandatory postmortems
- Executive escalation

---

Error Budget Burn Rate

Concept

Burn rate = rate of error budget consumption

Example:

• Monthly budget: 43.2 minutes (99.9% SLO)
• If consuming at 2x rate: Budget exhausted in 15 days
• If consuming at 10x rate: Budget exhausted in 3 days

Burn Rate Calculation

Burn rate = (actual_error_rate / allowed_error_rate)

Example:
SLO: 99.9% (0.1% allowed error rate)
Current error rate: 0.5%

Burn rate = 0.5% / 0.1% = 5x
Time to exhaust = 30 days / 5 = 6 days

Multi-Window Alerting

Alert on burn rate across multiple time windows:

Fast burn (1 hour window):

Burn rate > 14.4x → Exhausts budget in 2 days
Alert after 2 minutes
Severity: Critical (page immediately)

Moderate burn (6 hour window):

Burn rate > 6x → Exhausts budget in 5 days
Alert after 30 minutes
Severity: Warning (create ticket)

Slow burn (3 day window):

Burn rate > 1x → Exhausts budget by end of month
Alert after 6 hours
Severity: Info (monitor)

Implementation

Prometheus:

# Fast burn alert (1h window, 2m grace period)
- alert: ErrorBudgetFastBurn
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total[1h]))
    ) > (14.4 * 0.001)  # 14.4x burn rate for 99.9% SLO
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Fast error budget burn detected"
    description: "Error budget will be exhausted in 2 days at current rate"

# Slow burn alert (6h window, 30m grace period)
- alert: ErrorBudgetSlowBurn
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[6h]))
      /
      sum(rate(http_requests_total[6h]))
    ) > (6 * 0.001)  # 6x burn rate for 99.9% SLO
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Elevated error budget burn detected"

---

SLO Reporting

Dashboard Structure

Overall Health:

┌─────────────────────────────────────────┐
│  SLO Compliance: 99.92% ✅              │
│  Error Budget Remaining: 73% 🟢         │
│  Burn Rate: 0.8x 🟢                     │
└─────────────────────────────────────────┘

SLI Performance:

Latency p95: 420ms (Target: 500ms) ✅
Error Rate: 0.08% (Target: < 0.1%) ✅
Availability: 99.95% (Target: > 99.9%) ✅

Error Budget Trend:

Graph showing:
- Error budget consumption over time
- Burn rate spikes
- Incidents marked
- Deploy events overlaid

Monthly SLO Report

Template:

# SLO Report: October 2024

## Executive Summary
- ✅ All SLOs met this month
- 🟡 Latency SLO came close to violation (99.1% compliance)
- 3 incidents consumed 47% of error budget
- Error budget remaining: 53%

## SLO Performance

### Availability SLO: 99.9%
- Actual: 99.92%
- Status: ✅ Met
- Error budget consumed: 33%
- Downtime: 23 minutes (allowed: 43.2 minutes)

### Latency SLO: p95 < 500ms
- Actual p95: 445ms
- Status: ✅ Met
- Compliance: 99.1% (target: 99%)
- 0.9% of requests exceeded threshold

### Error Rate SLO: < 0.1%
- Actual: 0.05%
- Status: ✅ Met
- Error budget consumed: 50%

## Incidents

### Incident #1: Database Overload (Oct 5)
- Duration: 15 minutes
- Error budget consumed: 35%
- Root cause: Slow query after schema change
- Prevention: Added query review to deploy checklist

### Incident #2: API Gateway Timeout (Oct 12)
- Duration: 5 minutes
- Error budget consumed: 10%
- Root cause: Configuration error in load balancer
- Prevention: Automated configuration validation

### Incident #3: Upstream Service Degradation (Oct 20)
- Duration: 3 minutes
- Error budget consumed: 2%
- Root cause: Third-party API outage
- Prevention: Implemented circuit breaker

## Recommendations
1. Investigate latency near-miss (Oct 15-17)
2. Add automated rollback for database changes
3. Increase circuit breaker thresholds for third-party APIs
4. Consider tightening availability SLO to 99.95%

## Next Month's Focus
- Reduce p95 latency to 400ms
- Implement automated canary deployments
- Add synthetic monitoring for critical paths

---

SLA Structure

Components

Service Description:

The API Service provides RESTful endpoints for user management,
authentication, and data retrieval.

Covered Metrics:

- Availability: Service is reachable and returns valid responses
- Latency: Time from request to response
- Error Rate: Percentage of requests returning errors

SLA Targets:

Service commits to:
1. 99.9% monthly uptime
2. p95 API response time < 1 second
3. Error rate < 0.5%

Measurement:

Metrics calculated from server-side monitoring:
- Uptime: Successful health check probes / total probes
- Latency: Server-side request duration (p95)
- Errors: HTTP 5xx responses / total responses

Calculated monthly (first of month for previous month).

Exclusions:

SLA does not cover:
- Scheduled maintenance (with 7 days notice)
- Client-side network issues
- DDoS attacks or force majeure
- Beta/preview features
- Issues caused by customer misuse

Service Credits:

Monthly Uptime    | Service Credit
----------------  | --------------
< 99.9% (SLA)     | 10%
< 99.0%           | 25%
< 95.0%           | 50%

Claiming Credits:

Customer must:
1. Report violation within 30 days
2. Provide ticket numbers for support requests
3. Credits applied to next month's invoice
4. Credits do not exceed monthly fee

Example SLAs by Industry

E-commerce:

- 99.95% availability
- p95 page load < 2s
- p99 checkout < 5s
- Credits: 5% per 0.1% below target

Financial Services:

- 99.99% availability
- p99 transaction < 500ms
- Zero data loss
- Penalties: $10,000 per hour of downtime

Media/Content:

- 99.9% availability
- p95 video start < 3s
- No credit system (best effort latency)

---

Best Practices

1. SLOs Should Be User-Centric

❌ "Database queries < 100ms" ✅ "API response time p95 < 500ms"

2. Start Loose, Tighten Over Time

• Begin with achievable targets
• Build reliability culture
• Gradually raise bar

3. Fewer, Better SLOs

• 1-3 SLOs per service
• Focus on user impact
• Avoid SLO sprawl

4. SLAs More Conservative Than SLOs

Internal SLO: 99.95%
Customer SLA: 99.9%
Margin: 0.05% buffer

5. Make Error Budgets Actionable

• Define policies at different thresholds
• Empower teams to make tradeoffs
• Review in planning meetings

6. Document Everything

• How SLIs are measured
• Why targets were chosen
• Who owns each SLO
• How to interpret metrics

7. Review Regularly

• Monthly SLO reviews
• Quarterly SLO adjustments
• Annual SLA renegotiation

---

Common Pitfalls

1. Too Many SLOs

❌ 20 different SLOs per service ✅ 2-3 critical SLOs

2. Unrealistic Targets

❌ 99.999% for non-critical service ✅ 99.9% with room to improve

3. SLOs Without Error Budgets

❌ "Must always be 99.9%" ✅ "Budget for 0.1% errors"

4. No Consequences

❌ Missing SLO has no impact ✅ Deploy freeze when budget exhausted

5. SLA Equals SLO

❌ Promise exactly what you target ✅ SLA more conservative than SLO

6. Ignoring User Experience

❌ "Our servers are up 99.99%" ✅ "Users can complete actions 99.9% of the time"

7. Static Targets

❌ Set once, never revisit ✅ Quarterly reviews and adjustments

---

Tools and Automation

SLO Tracking Tools

Prometheus + Grafana:

• Use recording rules for SLIs
• Alert on burn rates
• Dashboard for compliance

Google Cloud SLO Monitoring:

• Built-in SLO tracking
• Automatic error budget calculation
• Integration with alerting

Datadog SLOs:

• UI for SLO definition
• Automatic burn rate alerts
• Status pages

Custom Tools:

• sloth: Generate Prometheus rules from SLO definitions
• slo-libsonnet: Jsonnet library for SLO monitoring

Example: Prometheus Recording Rules

groups:
  - name: sli_recording
    interval: 30s
    rules:
      # SLI: Request success rate
      - record: sli:request_success:ratio
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

      # SLI: Request latency (p95)
      - record: sli:request_latency:p95
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          )

      # Error budget burn rate (1h window)
      - record: slo:error_budget_burn_rate:1h
        expr: |
          (1 - sli:request_success:ratio) / 0.001