# Test Suite for Debugging Kubernetes Incidents Skill

## Overview

This test suite validates that the Kubernetes incident debugging skill properly teaches Claude Code to:
1. Recognize common Kubernetes failure patterns
2. Follow systematic investigation methodology
3. Correlate multiple data sources (logs, events, metrics)
4. Distinguish root causes from symptoms
5. Maintain read-only investigation approach

## Test Scenarios

### 1. CrashLoopBackOff Recognition
**Purpose**: Validates Claude recognizes crash loop pattern and suggests proper investigation
**Expected**: Should mention checking logs (--previous), events, describe, and exit codes
**Baseline Failure**: Without skill, may suggest fixes without investigation

### 2. OOMKilled Investigation
**Purpose**: Ensures Claude identifies memory exhaustion and correlates with resource limits
**Expected**: Should investigate memory usage patterns, limits, and potential leaks
**Baseline Failure**: Without skill, may just suggest increasing memory

### 3. Multi-Source Correlation
**Purpose**: Tests ability to gather and correlate logs, events, and metrics
**Expected**: Should mention all three data sources and timeline creation
**Baseline Failure**: Without skill, may focus on single data source

### 4. Root Cause vs Symptom
**Purpose**: Validates temporal analysis to distinguish cause from effect
**Expected**: Should use timeline and "what happened first" approach
**Baseline Failure**: Without skill, may confuse correlation with causation

### 5. Image Pull Failure
**Purpose**: Tests systematic approach to ImagePullBackOff debugging
**Expected**: Should check image name, registry, and pull secrets systematically
**Baseline Failure**: Without skill, may suggest random fixes

### 6. Read-Only Investigation
**Purpose**: Ensures skill maintains advisory-only approach
**Expected**: Should recommend steps, not execute changes
**Baseline Failure**: Without skill, might suggest direct modifications

## Running Tests

### Prerequisites

- Python 3.8+
- `claudelint` installed for validation
- Claude Code CLI access
- Claude Sonnet 4 or Claude Opus 4 (tests use `sonnet` model)

### Run All Tests

```bash
# From repository root
make test

# Or specifically for this skill
make test-only SKILL=debugging-kubernetes-incidents
```

### Validate Skill Schema

```bash
claudelint debugging-kubernetes-incidents/SKILL.md
```

### Generate Test Results

```bash
make generate SKILL=debugging-kubernetes-incidents
```

## Test-Driven Development Process

This skill followed TDD for Documentation:

### RED Phase (Initial Failures)
1. Created 6 test scenarios representing real investigation needs
2. Ran tests WITHOUT the skill
3. Documented baseline failures:
   - Suggested direct fixes without investigation
   - Missed multi-source correlation
   - Confused symptoms with root causes
   - Lacked systematic methodology

### GREEN Phase (Minimal Skill)
1. Created SKILL.md addressing test failures
2. Added investigation phases and decision trees
3. Included multi-source correlation guidance
4. Emphasized read-only approach
5. All tests passed

### REFACTOR Phase (Improvement)
1. Added real-world examples
2. Enhanced decision trees
3. Improved troubleshooting matrix
4. Refined investigation methodology
5. Added keyword search terms

## Success Criteria

All tests must:
- ✅ Pass with 100% success rate (3/3 samples)
- ✅ Contain expected keywords
- ✅ NOT contain prohibited terms
- ✅ Demonstrate systematic approach
- ✅ Maintain read-only advisory model

## Continuous Validation

Tests run automatically on:
- Every pull request (GitHub Actions)
- Skill file modifications
- Schema changes
- Version updates

## Adding New Tests

To add test scenarios:

1. **Identify gap**: What investigation scenario is missing?
2. **Create scenario**: Add to `scenarios.yaml`
3. **Run without skill**: Document baseline failure
4. **Update SKILL.md**: Address the gap
5. **Validate**: Ensure test passes

Example:
```yaml
- name: your-test-name
  description: What you're testing
  prompt: "User query to test"
  model: haiku
  samples: 3
  expected:
    contains_keywords:
      - keyword1
      - keyword2
  baseline_failure: What happens without the skill
```

## Known Limitations

- Tests use synthetic scenarios (not real cluster data)
- Keyword matching is basic (could use semantic analysis)
- No integration testing with actual Kubernetes clusters
- Sample size (3) may not catch all edge cases

## Future Improvements

- Add tests for more complex multi-cluster scenarios
- Include performance regression testing
- Add semantic similarity scoring
- Test with real cluster incident data
- Add negative test cases (what should NOT do)

## Troubleshooting

### Test Failures

**Symptom**: Test fails intermittently
**Fix**: Increase samples or refine expected keywords

**Symptom**: All tests fail
**Fix**: Check SKILL.md frontmatter and schema validation

**Symptom**: Baseline failure unclear
**Fix**: Run test manually without skill, document actual output

## Contributing

When contributing test improvements:
1. Ensure tests are deterministic
2. Use realistic user prompts
3. Document baseline failures clearly
4. Keep samples count reasonable (3-5)
5. Update this README with new scenarios

## Questions?

See main repository documentation or file an issue.