gh-lyndonkl-claude/skills/chain-spec-risk-metrics/resources/methodology.md

# Chain Spec Risk Metrics Methodology

Advanced techniques for complex multi-phase programs, novel risks, and sophisticated metric frameworks.

## Workflow

Copy this checklist and track your progress:

```
Chain Spec Risk Metrics Progress:
- [ ] Step 1: Gather initiative context
- [ ] Step 2: Write comprehensive specification
- [ ] Step 3: Conduct premortem and build risk register
- [ ] Step 4: Define success metrics and instrumentation
- [ ] Step 5: Validate completeness and deliver
```

**Step 1: Gather initiative context** - Collect goal, constraints, stakeholders, baseline. Use template questions plus assess complexity using [5. Complexity Decision Matrix](#5-complexity-decision-matrix).

**Step 2: Write comprehensive specification** - For multi-phase programs use [1. Advanced Specification Techniques](#1-advanced-specification-techniques) including phased rollout strategies and requirements traceability matrix.

**Step 3: Conduct premortem and build risk register** - Apply [2. Advanced Risk Assessment](#2-advanced-risk-assessment) methods including quantitative analysis, fault tree analysis, and premortem variations for comprehensive risk identification.

**Step 4: Define success metrics** - Use [3. Advanced Metrics Frameworks](#3-advanced-metrics-frameworks) such as HEART, AARRR, North Star, or SLI/SLO depending on initiative type.

**Step 5: Validate and deliver** - Ensure integration using [4. Integration Best Practices](#4-integration-best-practices) and check for [6. Anti-Patterns](#6-anti-patterns) before delivering.

## 1. Advanced Specification Techniques

### Multi-Phase Program Decomposition

**When**: Initiative is too large to execute in one phase (>6 months, >10 people, multiple systems).

**Approach**: Break into phases with clear value delivery at each stage.

**Phase Structure**:
- **Phase 0 (Discovery)**: Research, prototyping, validate assumptions
  - Deliverable: Technical spike, proof of concept, feasibility report
  - Metrics: Prototype success rate, assumption validation rate
- **Phase 1 (Foundation)**: Core infrastructure, no user-facing features yet
  - Deliverable: Platform/APIs deployed, instrumentation in place
  - Metrics: System stability, API latency, error rates
- **Phase 2 (Alpha)**: Limited rollout to internal users or pilot customers
  - Deliverable: Feature complete for target use case, internal feedback
  - Metrics: User activation, time-to-value, critical bugs
- **Phase 3 (Beta)**: Broader rollout, feature complete, gather feedback
  - Deliverable: Production-ready with known limitations
  - Metrics: Adoption rate, support tickets, performance under load
- **Phase 4 (GA)**: General availability, full feature set, scaled operations
  - Deliverable: Fully launched, documented, supported
  - Metrics: Market penetration, revenue, NPS, operational costs

**Phase Dependencies**:
- Document what each phase depends on (previous phase completion, external dependencies)
- Define phase gates (criteria to proceed to next phase)
- Include rollback plans if a phase fails

### Phased Rollout Strategies

**Canary Deployment**:
- Roll out to 1% → 10% → 50% → 100% of traffic/users
- Monitor metrics at each stage before expanding
- Automatic rollback if error rates spike

**Ring Deployment**:
- Ring 0: Internal employees (catch obvious bugs)
- Ring 1: Pilot customers (friendly users, willing to provide feedback)
- Ring 2: Early adopters (power users, high tolerance)
- Ring 3: General availability (all users)

**Feature Flags**:
- Deploy code but keep feature disabled
- Gradually enable for user segments
- A/B test impact before full rollout

**Geographic Rollout**:
- Region 1 (smallest/lowest risk) → Region 2 → Region 3 → Global
- Allows testing localization, compliance, performance in stages

### Requirements Traceability Matrix

For complex initiatives, map requirements → design → implementation → tests → risks → metrics.

| Requirement ID | Requirement | Design Doc | Implementation | Test Cases | Related Risks | Success Metric |
|----------------|-------------|------------|----------------|------------|---------------|----------------|
| REQ-001 | Auth via OAuth 2.0 | Design-Auth.md | auth-service/ | test/auth/ | R3, R7 | Auth success rate > 99.9% |
| REQ-002 | API p99 < 200ms | Design-Perf.md | gateway/ | test/perf/ | R1, R5 | p99 latency metric |

**Benefits**:
- Ensures nothing is forgotten (every requirement has tests, metrics, risk mitigation)
- Helps with impact analysis (if a risk materializes, which requirements are affected?)
- Useful for audit/compliance (trace from business need → implementation → validation)

## 2. Advanced Risk Assessment

### Quantitative Risk Analysis

**When**: High-stakes initiatives where "Low/Medium/High" risk scoring is insufficient.

**Approach**: Assign probabilities (%) and impact ($$) to risks, calculate expected loss.

**Example**:
- **Risk**: Database migration fails, requiring full rollback
- **Probability**: 15% (based on similar past migrations)
- **Impact**: $500K (3 eng-weeks × 10 engineers × $50K/year/eng + customer churn)
- **Expected Loss**: 15% × $500K = $75K
- **Mitigation Cost**: $30K (comprehensive testing, dry-run on prod snapshot)
- **Decision**: Invest $30K to mitigate (expected savings $45K)

**Three-Point Estimation** for uncertainty:
- **Optimistic**: Best-case scenario (10th percentile)
- **Most Likely**: Expected case (50th percentile)
- **Pessimistic**: Worst-case scenario (90th percentile)
- **Expected Value**: (Optimistic + 4×Most Likely + Pessimistic) / 6

### Fault Tree Analysis

**When**: Analyzing how multiple small failures combine to cause catastrophic outcome.

**Approach**: Work backward from catastrophic failure to root causes using logic gates.

**Example**: "Customer data breach"
```
[Customer Data Breach]
       ↓ (OR gate - any path causes breach)
    ┌──┴──┐
[Auth bypass] OR [DB exposed to internet]
    ↓ (AND gate - both required)
 ┌──┴──┐
[SQL injection] AND [No input validation]
```

**Insights**:
- Identify single points of failure (only one thing has to go wrong)
- Reveal defense-in-depth opportunities (add redundant protections)
- Prioritize mitigations (block root causes that appear in multiple paths)

### Bow-Tie Risk Diagrams

**When**: Complex risks with multiple causes and multiple consequences.

**Structure**:
```
[Causes] → [Preventive Controls] → [RISK EVENT] → [Mitigative Controls] → [Consequences]
```

**Example**: Risk = "Production database unavailable"

**Left side (Causes + Prevention)**:
- Cause 1: Hardware failure → Prevention: Redundant instances, health checks
- Cause 2: Human error (bad migration) → Prevention: Dry-run on snapshot, peer review
- Cause 3: DDoS attack → Prevention: Rate limiting, WAF

**Right side (Mitigation + Consequences)**:
- Consequence 1: Users can't access app → Mitigation: Read replica for degraded mode
- Consequence 2: Revenue loss → Mitigation: SLA credits, customer communication plan
- Consequence 3: Reputational damage → Mitigation: PR plan, status page transparency

**Benefits**: Shows full lifecycle of risk (how to prevent, how to respond if it happens anyway).

### Premortem Variations

**Reverse Premortem** ("It succeeded wildly - how?"):
- Identifies what must go RIGHT for success
- Reveals critical success factors often overlooked
- Example: "We succeeded because we secured executive sponsorship early and maintained it throughout"

**Pre-Parade** (best-case scenario):
- What would make this initiative exceed expectations?
- Identifies opportunities to amplify impact
- Example: "If we also integrate with Salesforce, we could unlock enterprise market"

**Lateral Premortem** (stakeholder-specific):
- Run separate premortems for each stakeholder group
- Engineering: "It failed because of technical reasons..."
- Product: "It failed because users didn't adopt it..."
- Operations: "It failed because we couldn't support it at scale..."

## 3. Advanced Metrics Frameworks

### HEART Framework (Google)

For user-facing products, track:
- **Happiness**: User satisfaction (NPS, CSAT surveys)
- **Engagement**: Level of user involvement (DAU/MAU, sessions/user)
- **Adoption**: New users accepting product (% of target users activated)
- **Retention**: Rate users come back (7-day/30-day retention)
- **Task Success**: Efficiency completing tasks (completion rate, time on task, error rate)

**Application**:
- Leading: Adoption rate, task success rate
- Lagging: Retention, engagement over time
- Counter-metric: Happiness (don't sacrifice UX for engagement)

### AARRR Pirate Metrics

For growth-focused initiatives:
- **Acquisition**: Users discover product (traffic sources, signup rate)
- **Activation**: Users have good first experience (onboarding completion, time-to-aha)
- **Retention**: Users return (D1/D7/D30 retention)
- **Revenue**: Users pay (conversion rate, ARPU, LTV)
- **Referral**: Users bring others (viral coefficient, NPS)

**Application**:
- Identify bottleneck stage (where most users drop off)
- Focus initiative on improving that stage
- Track funnel conversion at each stage

### North Star + Supporting Metrics

**North Star Metric**: Single metric that best captures value delivered to customers.
- Examples: Weekly active users (social app), time-to-insight (analytics), API calls/week (platform)

**Supporting Metrics**: Inputs that drive the North Star.
- Example NSM: Weekly Active Users
  - Supporting: New user signups, activation rate, feature engagement, retention rate

**Application**:
- All initiatives tie back to moving North Star or supporting metrics
- Prevents metric myopia (optimizing one metric at expense of others)
- Aligns team around common goal

### SLI/SLO/SLA Framework

For reliability-focused initiatives:
- **SLI (Service Level Indicator)**: What you measure (latency, error rate, throughput)
- **SLO (Service Level Objective)**: Target for SLI (p99 latency < 200ms, error rate < 0.1%)
- **SLA (Service Level Agreement)**: Consequence if SLO not met (customer credits, escalation)

**Error Budget**:
- If SLO is 99.9% uptime, error budget is 0.1% downtime (43 minutes/month)
- Can "spend" error budget on risky deployments
- If budget exhausted, halt feature work and focus on reliability

**Application**:
- Define SLIs/SLOs for each major component
- Track burn rate (how fast you're consuming error budget)
- Use as gate for deployment (don't deploy if error budget exhausted)

### Metrics Decomposition Trees

**When**: Complex metric needs to be broken down into actionable components.

**Example**: Increase revenue
```
Revenue
├─ New Customer Revenue
│  ├─ New Customers (leads × conversion rate)
│  └─ Average Deal Size (features × pricing tier)
└─ Existing Customer Revenue
   ├─ Expansion (upsell rate × expansion ARR)
   └─ Retention (renewal rate × existing ARR)
```

**Application**:
- Identify which leaf nodes to focus on
- Each leaf becomes a metric to track
- Reveals non-obvious leverage points (e.g., increasing renewal rate might have bigger impact than new customer acquisition)

## 4. Integration Best Practices

### Specs → Risks Mapping

**Principle**: Every major specification decision should have corresponding risks identified.

**Example**:
- **Spec**: "Use MongoDB for user data storage"
- **Risks**:
  - R1: MongoDB performance degrades above 10M documents (mitigation: sharding strategy)
  - R2: Team lacks MongoDB expertise (mitigation: training, hire consultant)
  - R3: Data model changes require migration (mitigation: schema versioning)

**Implementation**:
- Review each spec section and ask "What could go wrong with this choice?"
- Ensure alternative approaches are considered with their risks
- Document why chosen approach is best despite risks

### Risks → Metrics Mapping

**Principle**: Each high-impact risk should have a metric that detects if it's materializing.

**Example**:
- **Risk**: "Database performance degrades under load"
- **Metrics**:
  - Leading: Query time p95 (early warning before user impact)
  - Lagging: User-reported latency complaints
  - Counter-metric: Don't over-optimize for read speed at expense of write consistency

**Implementation**:
- For each risk score 6-9, define early warning metric
- Set up alerts/dashboards to monitor these metrics
- Define escalation thresholds (when to invoke mitigation plan)

### Metrics → Specs Mapping

**Principle**: Specifications should include instrumentation to enable metric collection.

**Example**:
- **Metric**: "API p99 latency < 200ms"
- **Spec Requirements**:
  - Distributed tracing (OpenTelemetry) for end-to-end latency
  - Per-endpoint latency bucketing (identify slow endpoints)
  - Client-side RUM (real user monitoring) for user-perceived latency

**Implementation**:
- Review metrics and ask "What instrumentation is needed?"
- Add observability requirements to spec (logging, metrics, tracing)
- Include instrumentation in acceptance criteria

### Continuous Validation Loop

**Pattern**: Spec → Implement → Measure → Validate Risks → Update Spec

**Steps**:
1. **Initial Spec**: Document approach, risks, metrics
2. **Phase 1 Implementation**: Build and deploy
3. **Measure Reality**: Collect actual metrics vs. targets
4. **Validate Risk Mitigations**: Did mitigations work? New risks emerged?
5. **Update Spec**: Revise for next phase based on learnings

**Example**:
- **Phase 1 Spec**: Expected 10K req/s with single instance
- **Reality**: Actual 3K req/s (bottleneck in DB queries)
- **Risk Update**: Add R8: Query optimization needed for target load
- **Metric Update**: Add query execution time to dashboards
- **Phase 2 Spec**: Refactor queries, add read replicas, target 12K req/s

## 5. Complexity Decision Matrix

Use this matrix to decide when to use this skill vs. simpler approaches:

| Initiative Characteristics | Recommended Approach |
|---------------------------|----------------------|
| **Low Stakes** (< 1 eng-month, reversible, no user impact) | Lightweight spec, simple checklist, skip formal risk register |
| **Medium Stakes** (1-3 months, some teams, moderate impact) | Use template.md: Full spec + premortem + 3-5 metrics |
| **High Stakes** (3-6 months, cross-team, high impact) | Use template.md + methodology: Quantitative risk analysis, comprehensive metrics |
| **Strategic** (6+ months, company-level, existential) | Use methodology + external review: Fault tree, SLI/SLOs, continuous validation |

**Heuristics**:
- If failure would cost >$100K or 6+ eng-months, use full methodology
- If initiative affects >1000 users or >3 teams, use at least template
- If uncertainty is high, invest more in risk analysis and phased rollout
- If metrics are complex or novel, use advanced frameworks (HEART, North Star)

## 6. Anti-Patterns

**Spec Only (No Risks/Metrics)**:
- Symptom: Detailed spec but no discussion of what could go wrong or how to measure success
- Fix: Run quick premortem (15 min), define 3 must-track metrics minimum

**Risk Theater (Long List, No Mitigations)**:
- Symptom: 50+ risks identified but no prioritization or mitigation plans
- Fix: Score risks, focus on top 10, assign owners and mitigations

**Vanity Metrics (Measures Activity Not Outcomes)**:
- Symptom: Tracking "features shipped" instead of "user value delivered"
- Fix: For each metric ask "If this goes up, are users/business better off?"

**Plan and Forget (No Updates)**:
- Symptom: Beautiful spec/risk/metrics doc created then never referenced
- Fix: Schedule monthly reviews, update risks/metrics, track in team rituals

**Premature Precision (Overconfident Estimates)**:
- Symptom: "Migration will take exactly 47 days and cost $487K"
- Fix: Use ranges (30-60 days, $400-600K), state confidence levels, build in buffer

**Analysis Paralysis (Perfect Planning)**:
- Symptom: Spent 3 months planning, haven't started building
- Fix: Time-box planning (1-2 weeks for most initiatives), embrace uncertainty, learn by doing

## 7. Templates for Common Initiative Types

### Migration (System/Platform/Data)
- **Spec Focus**: Current vs. target architecture, migration path, rollback plan, validation
- **Risk Focus**: Data loss, downtime, performance regression, failed migration
- **Metrics Focus**: Migration progress %, data integrity, system performance, rollback capability

### Launch (Product/Feature)
- **Spec Focus**: User stories, UX flows, technical design, launch checklist
- **Risk Focus**: Low adoption, technical bugs, scalability issues, competitive response
- **Metrics Focus**: Activation, engagement, retention, revenue impact, support tickets

### Infrastructure Change
- **Spec Focus**: Architecture diagram, capacity planning, runbooks, disaster recovery
- **Risk Focus**: Outages, cost overruns, security vulnerabilities, operational complexity
- **Metrics Focus**: Uptime, latency, costs, incident count, MTTR

### Process Change (Organizational)
- **Spec Focus**: Current vs. new process, roles/responsibilities, training plan, timeline
- **Risk Focus**: Adoption resistance, productivity drop, key person dependency, communication gaps
- **Metrics Focus**: Process compliance, cycle time, employee satisfaction, quality metrics

For complete worked example, see [examples/microservices-migration.md](../examples/microservices-migration.md).