gh-yaleh-meta-cc-claude/skills/methodology-bootstrapping/examples/testing-methodology.md

Testing Methodology Example

Experiment: bootstrap-002-test-strategy Domain: Testing Strategy Iterations: 6 Final Coverage: 72.5% Patterns: 8 Tools: 3 Speedup: 5x

Complete walkthrough of applying BAIME to create testing methodology.

---

Iteration 0: Baseline (60 min)

Observations

Initial State:

• Coverage: 72.1%
• Tests: 590 total
• No systematic approach
• Ad-hoc test writing (15-25 min per test)

Problems Identified:

1. No clear test patterns
2. Unclear which functions to test first
3. Repetitive test setup code
4. No automation for coverage analysis
5. Inconsistent test quality

Baseline Metrics:

V_instance = 0.70 (coverage 72.1/75 × 0.5 + other metrics)
V_meta = 0.00 (no patterns yet)

---

Iteration 1: Core Patterns (90 min)

Created Patterns

Pattern 1: Table-Driven Tests

func TestFunction(t *testing.T) {
    tests := []struct {
        name string
        input int
        want int
    }{
        {"zero", 0, 0},
        {"positive", 5, 25},
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := Function(tt.input)
            if got != tt.want {
                t.Errorf("got %v, want %v", got, tt.want)
            }
        })
    }
}

• Time: 12 min per test (vs 18 min manual)
• Applied: 3 test functions
• Result: All passed

Pattern 2: Error Path Testing

tests := []struct {
    name string
    input Type
    wantErr bool
    errMsg string
}{
    {"nil input", nil, true, "cannot be nil"},
    {"empty", Type{}, true, "empty"},
}

• Time: 14 min per test
• Applied: 2 test functions
• Result: Found 1 bug (nil handling missing)

Results

Metrics:

• Tests added: 5
• Coverage: 72.1% → 72.8% (+0.7%)
• V_instance = 0.72
• V_meta = 0.25 (2/8 patterns)

---

Iteration 2: Expand & Automate (90 min)

New Patterns

Pattern 3: CLI Command Testing Pattern 4: Integration Tests Pattern 5: Test Helpers

First Automation Tool

Tool: Coverage Gap Analyzer

#!/bin/bash
go tool cover -func=coverage.out |
  grep "0.0%" |
  awk '{print $1, $2}' |
  sort

Speedup: 15 min manual → 30 sec automated (30x) ROI: 30 min to create, used 12 times = 180 min saved = 6x

Results

Metrics:

• Patterns: 5 total
• Tests added: 8
• Coverage: 72.8% → 73.5% (+0.7%)
• V_instance = 0.76
• V_meta = 0.42 (5/8 patterns, automation started)

---

Iteration 3: CLI Focus (75 min)

Expanded Patterns

Pattern 6: Global Flag Testing Pattern 7: Fixture Patterns

Results

Metrics:

• Patterns: 7 total
• Tests added: 12 (CLI-focused)
• Coverage: 73.5% → 74.8% (+1.3%)
• V_instance = 0.81 ✓ (exceeded target!)
• V_meta = 0.61 (7/8 patterns, 1 tool)

---

Iteration 4: Meta-Layer Push (90 min)

Completed Pattern Library

Pattern 8: Dependency Injection (Mocking)

Added Automation Tools

Tool 2: Test Generator

./scripts/generate-test.sh FunctionName --pattern table-driven

• Speedup: 10 min → 1 min (10x)
• ROI: 1 hour to create, used 8 times = 72 min saved = 1.2x

Tool 3: Methodology Guide Generator

• Auto-generates testing guide from patterns
• Speedup: 6 hours manual → 48 min automated (7.5x)

Results

Metrics:

• Patterns: 8 total (complete)
• Tests added: 6
• Coverage: 74.8% → 75.2% (+0.4%)
• V_instance = 0.82 ✓
• V_meta = 0.67 (8/8 patterns, 3 tools, ~75% complete)

---

Iteration 5: Refinement (60 min)

Activities

• Refined pattern documentation
• Tested transferability (Python, Rust, TypeScript)
• Measured cross-language applicability
• Consolidated examples

Results

Metrics:

• Patterns: 8 (refined, no new)
• Tests added: 4
• Coverage: 75.2% → 75.6% (+0.4%)
• V_instance = 0.84 ✓ (stable)
• V_meta = 0.78 (close to convergence!)

---

Iteration 6: Convergence (45 min)

Activities

• Final documentation polish
• Complete transferability guide
• Measure automation effectiveness
• Validate dual convergence

Results

Metrics:

• Patterns: 8 (final)
• Tests: 612 total (+22 from start)
• Coverage: 75.6% → 75.8% (+0.2%)
• V_instance = 0.85 ✓ (2 consecutive ≥ 0.80)
• V_meta = 0.82 ✓ (2 consecutive ≥ 0.80)

CONVERGED! ✅

---

Final Methodology

8 Patterns Documented

1. Unit Test Pattern (8 min)
2. Table-Driven Pattern (12 min)
3. Integration Test Pattern (18 min)
4. Error Path Pattern (14 min)
5. Test Helper Pattern (5 min)
6. Dependency Injection Pattern (22 min)
7. CLI Command Pattern (13 min)
8. Global Flag Pattern (11 min)

Average: 12.9 min per test (vs 20 min ad-hoc) Speedup: 1.55x from patterns alone

3 Automation Tools

1. Coverage Gap Analyzer: 30x speedup
2. Test Generator: 10x speedup
3. Methodology Guide Generator: 7.5x speedup

Combined Speedup: 5x overall

Transferability

• Go: 100% (native)
• Python: 90% (pytest compatible)
• Rust: 85% (rstest compatible)
• TypeScript: 85% (Jest compatible)
• Overall: 90% transferable

---

Key Learnings

What Worked Well

1. Strong Iteration 0: Comprehensive baseline saved time later
2. Focus on CLI: High-impact area (cmd/ package 55% → 73%)
3. Early automation: Tool ROI paid off quickly
4. Pattern consolidation: Stopped at 8 patterns (not bloated)

Challenges

1. Coverage plateaued: Hard to improve beyond 75%
2. Tool creation time: Automation took longer than expected (1-2 hours each)
3. Transferability testing: Required extra time to validate cross-language

Would Do Differently

1. Start automation earlier (Iteration 1 vs Iteration 2)
2. Limit pattern count from start (set 8 as max)
3. Test transferability incrementally (don't wait until end)

---

Replication Guide

To Apply to Your Project

Week 1: Foundation (Iterations 0-2)

# Day 1: Baseline
go test -cover ./...
# Document current coverage and problems

# Day 2-3: Core patterns
# Create 2-3 patterns addressing top problems
# Test on real examples

# Day 4-5: Automation
# Create coverage gap analyzer
# Measure speedup

Week 2: Expansion (Iterations 3-4)

# Day 1-2: Additional patterns
# Expand to 6-8 patterns total

# Day 3-4: More automation
# Create test generator
# Calculate ROI

# Day 5: V_instance convergence
# Ensure metrics meet targets

Week 3: Meta-Layer (Iterations 5-6)

# Day 1-2: Refinement
# Polish documentation
# Test transferability

# Day 3-4: Final automation
# Complete tool suite
# Measure effectiveness

# Day 5: Validation
# Confirm dual convergence
# Prepare production documentation

Customization by Project Size

Small Project (<10k LOC):

• 4 iterations sufficient
• 5-6 patterns
• 2 automation tools
• Total time: ~6 hours

Medium Project (10-50k LOC):

• 5-6 iterations (standard)
• 6-8 patterns
• 3 automation tools
• Total time: ~8-10 hours

Large Project (>50k LOC):

• 6-8 iterations
• 8-10 patterns
• 4-5 automation tools
• Total time: ~12-15 hours

---

Source: Bootstrap-002 Test Strategy Development Status: Production-ready, dual convergence achieved Total Time: 7.5 hours (6 iterations × 75 min avg) ROI: 5x speedup, 90% transferable