348 lines
7.2 KiB
Markdown
348 lines
7.2 KiB
Markdown
# Testing Methodology Example
|
||
|
||
**Experiment**: bootstrap-002-test-strategy
|
||
**Domain**: Testing Strategy
|
||
**Iterations**: 6
|
||
**Final Coverage**: 72.5%
|
||
**Patterns**: 8
|
||
**Tools**: 3
|
||
**Speedup**: 5x
|
||
|
||
Complete walkthrough of applying BAIME to create testing methodology.
|
||
|
||
---
|
||
|
||
## Iteration 0: Baseline (60 min)
|
||
|
||
### Observations
|
||
|
||
**Initial State**:
|
||
- Coverage: 72.1%
|
||
- Tests: 590 total
|
||
- No systematic approach
|
||
- Ad-hoc test writing (15-25 min per test)
|
||
|
||
**Problems Identified**:
|
||
1. No clear test patterns
|
||
2. Unclear which functions to test first
|
||
3. Repetitive test setup code
|
||
4. No automation for coverage analysis
|
||
5. Inconsistent test quality
|
||
|
||
**Baseline Metrics**:
|
||
```
|
||
V_instance = 0.70 (coverage 72.1/75 × 0.5 + other metrics)
|
||
V_meta = 0.00 (no patterns yet)
|
||
```
|
||
|
||
---
|
||
|
||
## Iteration 1: Core Patterns (90 min)
|
||
|
||
### Created Patterns
|
||
|
||
**Pattern 1: Table-Driven Tests**
|
||
```go
|
||
func TestFunction(t *testing.T) {
|
||
tests := []struct {
|
||
name string
|
||
input int
|
||
want int
|
||
}{
|
||
{"zero", 0, 0},
|
||
{"positive", 5, 25},
|
||
}
|
||
for _, tt := range tests {
|
||
t.Run(tt.name, func(t *testing.T) {
|
||
got := Function(tt.input)
|
||
if got != tt.want {
|
||
t.Errorf("got %v, want %v", got, tt.want)
|
||
}
|
||
})
|
||
}
|
||
}
|
||
```
|
||
- **Time**: 12 min per test (vs 18 min manual)
|
||
- **Applied**: 3 test functions
|
||
- **Result**: All passed
|
||
|
||
**Pattern 2: Error Path Testing**
|
||
```go
|
||
tests := []struct {
|
||
name string
|
||
input Type
|
||
wantErr bool
|
||
errMsg string
|
||
}{
|
||
{"nil input", nil, true, "cannot be nil"},
|
||
{"empty", Type{}, true, "empty"},
|
||
}
|
||
```
|
||
- **Time**: 14 min per test
|
||
- **Applied**: 2 test functions
|
||
- **Result**: Found 1 bug (nil handling missing)
|
||
|
||
### Results
|
||
|
||
**Metrics**:
|
||
- Tests added: 5
|
||
- Coverage: 72.1% → 72.8% (+0.7%)
|
||
- V_instance = 0.72
|
||
- V_meta = 0.25 (2/8 patterns)
|
||
|
||
---
|
||
|
||
## Iteration 2: Expand & Automate (90 min)
|
||
|
||
### New Patterns
|
||
|
||
**Pattern 3: CLI Command Testing**
|
||
**Pattern 4: Integration Tests**
|
||
**Pattern 5: Test Helpers**
|
||
|
||
### First Automation Tool
|
||
|
||
**Tool**: Coverage Gap Analyzer
|
||
```bash
|
||
#!/bin/bash
|
||
go tool cover -func=coverage.out |
|
||
grep "0.0%" |
|
||
awk '{print $1, $2}' |
|
||
sort
|
||
```
|
||
|
||
**Speedup**: 15 min manual → 30 sec automated (30x)
|
||
**ROI**: 30 min to create, used 12 times = 180 min saved = 6x
|
||
|
||
### Results
|
||
|
||
**Metrics**:
|
||
- Patterns: 5 total
|
||
- Tests added: 8
|
||
- Coverage: 72.8% → 73.5% (+0.7%)
|
||
- V_instance = 0.76
|
||
- V_meta = 0.42 (5/8 patterns, automation started)
|
||
|
||
---
|
||
|
||
## Iteration 3: CLI Focus (75 min)
|
||
|
||
### Expanded Patterns
|
||
|
||
**Pattern 6: Global Flag Testing**
|
||
**Pattern 7: Fixture Patterns**
|
||
|
||
### Results
|
||
|
||
**Metrics**:
|
||
- Patterns: 7 total
|
||
- Tests added: 12 (CLI-focused)
|
||
- Coverage: 73.5% → 74.8% (+1.3%)
|
||
- **V_instance = 0.81** ✓ (exceeded target!)
|
||
- V_meta = 0.61 (7/8 patterns, 1 tool)
|
||
|
||
---
|
||
|
||
## Iteration 4: Meta-Layer Push (90 min)
|
||
|
||
### Completed Pattern Library
|
||
|
||
**Pattern 8: Dependency Injection (Mocking)**
|
||
|
||
### Added Automation Tools
|
||
|
||
**Tool 2**: Test Generator
|
||
```bash
|
||
./scripts/generate-test.sh FunctionName --pattern table-driven
|
||
```
|
||
- **Speedup**: 10 min → 1 min (10x)
|
||
- **ROI**: 1 hour to create, used 8 times = 72 min saved = 1.2x
|
||
|
||
**Tool 3**: Methodology Guide Generator
|
||
- Auto-generates testing guide from patterns
|
||
- **Speedup**: 6 hours manual → 48 min automated (7.5x)
|
||
|
||
### Results
|
||
|
||
**Metrics**:
|
||
- Patterns: 8 total (complete)
|
||
- Tests added: 6
|
||
- Coverage: 74.8% → 75.2% (+0.4%)
|
||
- V_instance = 0.82 ✓
|
||
- **V_meta = 0.67** (8/8 patterns, 3 tools, ~75% complete)
|
||
|
||
---
|
||
|
||
## Iteration 5: Refinement (60 min)
|
||
|
||
### Activities
|
||
|
||
- Refined pattern documentation
|
||
- Tested transferability (Python, Rust, TypeScript)
|
||
- Measured cross-language applicability
|
||
- Consolidated examples
|
||
|
||
### Results
|
||
|
||
**Metrics**:
|
||
- Patterns: 8 (refined, no new)
|
||
- Tests added: 4
|
||
- Coverage: 75.2% → 75.6% (+0.4%)
|
||
- V_instance = 0.84 ✓ (stable)
|
||
- **V_meta = 0.78** (close to convergence!)
|
||
|
||
---
|
||
|
||
## Iteration 6: Convergence (45 min)
|
||
|
||
### Activities
|
||
|
||
- Final documentation polish
|
||
- Complete transferability guide
|
||
- Measure automation effectiveness
|
||
- Validate dual convergence
|
||
|
||
### Results
|
||
|
||
**Metrics**:
|
||
- Patterns: 8 (final)
|
||
- Tests: 612 total (+22 from start)
|
||
- Coverage: 75.6% → 75.8% (+0.2%)
|
||
- **V_instance = 0.85** ✓ (2 consecutive ≥ 0.80)
|
||
- **V_meta = 0.82** ✓ (2 consecutive ≥ 0.80)
|
||
|
||
**CONVERGED!** ✅
|
||
|
||
---
|
||
|
||
## Final Methodology
|
||
|
||
### 8 Patterns Documented
|
||
|
||
1. Unit Test Pattern (8 min)
|
||
2. Table-Driven Pattern (12 min)
|
||
3. Integration Test Pattern (18 min)
|
||
4. Error Path Pattern (14 min)
|
||
5. Test Helper Pattern (5 min)
|
||
6. Dependency Injection Pattern (22 min)
|
||
7. CLI Command Pattern (13 min)
|
||
8. Global Flag Pattern (11 min)
|
||
|
||
**Average**: 12.9 min per test (vs 20 min ad-hoc)
|
||
**Speedup**: 1.55x from patterns alone
|
||
|
||
### 3 Automation Tools
|
||
|
||
1. **Coverage Gap Analyzer**: 30x speedup
|
||
2. **Test Generator**: 10x speedup
|
||
3. **Methodology Guide Generator**: 7.5x speedup
|
||
|
||
**Combined Speedup**: 5x overall
|
||
|
||
### Transferability
|
||
|
||
- **Go**: 100% (native)
|
||
- **Python**: 90% (pytest compatible)
|
||
- **Rust**: 85% (rstest compatible)
|
||
- **TypeScript**: 85% (Jest compatible)
|
||
- **Overall**: 90% transferable
|
||
|
||
---
|
||
|
||
## Key Learnings
|
||
|
||
### What Worked Well
|
||
|
||
1. **Strong Iteration 0**: Comprehensive baseline saved time later
|
||
2. **Focus on CLI**: High-impact area (cmd/ package 55% → 73%)
|
||
3. **Early automation**: Tool ROI paid off quickly
|
||
4. **Pattern consolidation**: Stopped at 8 patterns (not bloated)
|
||
|
||
### Challenges
|
||
|
||
1. **Coverage plateaued**: Hard to improve beyond 75%
|
||
2. **Tool creation time**: Automation took longer than expected (1-2 hours each)
|
||
3. **Transferability testing**: Required extra time to validate cross-language
|
||
|
||
### Would Do Differently
|
||
|
||
1. **Start automation earlier** (Iteration 1 vs Iteration 2)
|
||
2. **Limit pattern count** from start (set 8 as max)
|
||
3. **Test transferability incrementally** (don't wait until end)
|
||
|
||
---
|
||
|
||
## Replication Guide
|
||
|
||
### To Apply to Your Project
|
||
|
||
**Week 1: Foundation (Iterations 0-2)**
|
||
```bash
|
||
# Day 1: Baseline
|
||
go test -cover ./...
|
||
# Document current coverage and problems
|
||
|
||
# Day 2-3: Core patterns
|
||
# Create 2-3 patterns addressing top problems
|
||
# Test on real examples
|
||
|
||
# Day 4-5: Automation
|
||
# Create coverage gap analyzer
|
||
# Measure speedup
|
||
```
|
||
|
||
**Week 2: Expansion (Iterations 3-4)**
|
||
```bash
|
||
# Day 1-2: Additional patterns
|
||
# Expand to 6-8 patterns total
|
||
|
||
# Day 3-4: More automation
|
||
# Create test generator
|
||
# Calculate ROI
|
||
|
||
# Day 5: V_instance convergence
|
||
# Ensure metrics meet targets
|
||
```
|
||
|
||
**Week 3: Meta-Layer (Iterations 5-6)**
|
||
```bash
|
||
# Day 1-2: Refinement
|
||
# Polish documentation
|
||
# Test transferability
|
||
|
||
# Day 3-4: Final automation
|
||
# Complete tool suite
|
||
# Measure effectiveness
|
||
|
||
# Day 5: Validation
|
||
# Confirm dual convergence
|
||
# Prepare production documentation
|
||
```
|
||
|
||
### Customization by Project Size
|
||
|
||
**Small Project (<10k LOC)**:
|
||
- 4 iterations sufficient
|
||
- 5-6 patterns
|
||
- 2 automation tools
|
||
- Total time: ~6 hours
|
||
|
||
**Medium Project (10-50k LOC)**:
|
||
- 5-6 iterations (standard)
|
||
- 6-8 patterns
|
||
- 3 automation tools
|
||
- Total time: ~8-10 hours
|
||
|
||
**Large Project (>50k LOC)**:
|
||
- 6-8 iterations
|
||
- 8-10 patterns
|
||
- 4-5 automation tools
|
||
- Total time: ~12-15 hours
|
||
|
||
---
|
||
|
||
**Source**: Bootstrap-002 Test Strategy Development
|
||
**Status**: Production-ready, dual convergence achieved
|
||
**Total Time**: 7.5 hours (6 iterations × 75 min avg)
|
||
**ROI**: 5x speedup, 90% transferable
|