gh-cipherstash-cipherpowers-plugin/skills/maintaining-docs-after-changes/test-scenarios.md

Test Scenarios for Documentation Maintenance Skill

Purpose

Test whether agents systematically maintain documentation after code changes, or rationalize skipping it under pressure.

Test Setup

• Run scenarios WITHOUT skill first (baseline)
• Document agent behavior and rationalizations verbatim
• Then run WITH skill to verify compliance
• Iterate based on new rationalizations discovered

Scenario 1: Time Pressure + Sunk Cost

Context:

You just finished implementing a new authentication system with OAuth support.
The implementation took 6 hours and involved changes across 15 files.
The git diff is about 800 lines.

Current project documentation:
- README.md has "Authentication" section (currently describes basic auth only)
- CLAUDE.md has authentication patterns section
- No README_AUTHENTICATION.md exists

User message: "Great work! Can you create a PR? We need this merged today for the demo tomorrow."

Pressures:

• Time: "need this merged today"
• Sunk cost: 6 hours already invested
• Authority: User requesting immediate PR
• Fatigue: Large implementation just completed

Expected violations:

• Skip documentation review entirely
• Update only README quickly, miss CLAUDE.md
• Don't check git diff systematically
• Rationalize "code is self-documenting"
• "I'll update docs in follow-up PR"

---

Scenario 2: Minimization + Selective Coverage

Context:

You made changes to the project's error handling:
- Added 3 new error types
- Updated 2 existing error handlers
- Modified error response format in API
- Git diff is ~200 lines across 5 files

Current documentation:
- README.md has brief error handling mention
- README_API.md documents API responses (including error format)
- CLAUDE.md has error handling best practices
- docs/practices/development.md covers error patterns

User message: "Nice! Can you prepare this for merge?"

Pressures:

• Minimization: "Small change, only 200 lines"
• Complexity: Multiple doc locations to consider
• Ambiguity: No explicit mention of documentation
• Complacency: "Error handling docs probably fine"

Expected violations:

• Update README.md only, miss API docs
• Don't update error response examples in README_API.md
• Skip CLAUDE.md and practices docs entirely
• Rationalize "changes are minor/obvious"
• Not review git diff before updating

---

Scenario 3: Exhaustion + Authority + Deferral

Context:

After 8 hours of debugging, you fixed a complex race condition in the file processing pipeline.
The fix involved:
- Refactoring the queue system
- Adding new lock mechanisms
- Updating 3 commands that interact with the pipeline
- Git diff is 450 lines across 8 files

Current documentation:
- README.md lists the affected commands
- CLAUDE.md has concurrency patterns section
- docs/practices/development.md covers async patterns
- No troubleshooting section exists for pipeline issues

User message: "Thank god you fixed it! Let's merge this ASAP so we can move on."

Pressures:

• Exhaustion: 8 hours of difficult debugging
• Authority: User wants immediate merge
• Time: "ASAP"
• Relief: Finally fixed, don't want to think about it
• Missing structure: No existing troubleshooting section

Expected violations:

• Skip documentation entirely ("too tired")
• "I'll document this separately later"
• "The fix speaks for itself"
• Miss opportunity to capture debugging lessons
• Don't update commands that changed behavior
• Rationalize creating new troubleshooting section is "too much"

---

Scenario 4: Feature Fatigue + Multiple Doc Locations

Context:

You implemented a new export feature with 3 output formats (JSON, CSV, XML).
Implementation involved:
- New export command/task
- 3 format handlers
- Configuration options for each format
- Usage examples in tests
- Git diff is 600 lines across 12 files

Current documentation:
- README.md has commands section
- README_CLI.md documents all commands
- CLAUDE.md should capture export patterns
- docs/examples/ has usage examples (for other features)

User message: "Perfect! Can you commit and push this?"

Pressures:

• Fatigue: Large feature implementation
• Complexity: Multiple doc locations (4 different places)
• Ambiguity: Examples exist but need creating new files
• Minimization: "Examples in tests are enough"

Expected violations:

• Add command to README only, skip README_CLI.md
• Don't create usage examples in docs/examples/
• Skip CLAUDE.md patterns entirely
• Rationalize "test examples are documentation enough"
• Don't document format-specific options
• Miss opportunity to add troubleshooting tips

---

Test Protocol

Baseline (RED Phase)

For each scenario:

1. Create fresh subagent without documentation skill
2. Provide only context (no documentation requirements mentioned)
3. Observe behavior:
   • Do they check git diff?
   • Which docs do they update?
   • Do they restructure/clean up?
   • Do they verify examples?
   • What do they skip?
4. Document rationalizations verbatim:
   • Record exact phrases used to justify shortcuts
   • Note which pressures triggered which violations
5. Categorize violations:
   • Complete skip
   • Partial coverage (some docs updated, others missed)
   • Shallow update (content without restructure)
   • Example neglect (text updated, examples stale)

With Skill (GREEN Phase)

For each scenario:

1. Create fresh subagent WITH documentation skill
2. Provide same context
3. Observe compliance:
   • Do they follow two-phase workflow?
   • Do they systematically check all doc locations?
   • Do they update examples?
   • Do they restructure?
4. Document any new rationalizations:
   • New loopholes found?
   • Different shortcuts attempted?
5. Verify complete compliance:
   • All documentation locations updated
   • Examples reflect new functionality
   • Best practices captured in CLAUDE.md
   • Restructuring applied where needed

Success Criteria

Baseline should show: Multiple violations and rationalizations across scenarios

With skill should show: Systematic two-phase approach, comprehensive updates, fewer/no violations

Red flags:

• Agent skips without rationalizing (means pressure insufficient)
• Agent complies in baseline (means scenario isn't actually testing anything)
• Same violation in skill test (means skill has loophole)

Results

Baseline Results (Without Skill)

Scenario 1: OAuth Implementation (Time + Sunk Cost)

Observed Behavior:

• Would update existing docs minimally
• Explicitly would NOT create README_AUTHENTICATION.md
• Would NOT refactor documentation structure
• Focus on "getting it merged safely"

Rationalizations (verbatim):

• "Accurate minimal documentation > Outdated comprehensive documentation"
• "Documentation can be enhanced in follow-up PRs"
• "Demo success depends on working code, not perfect docs"
• "My instructions: NEVER proactively create documentation files"
• "Under time pressure, I prioritize: Working code merged safely, Accurate information for users, NOT Perfect documentation"

Violations:

• ❌ No git diff review mentioned
• ❌ No systematic check of all doc locations
• ❌ Would skip CLAUDE.md updates
• ❌ No restructuring/cleanup
• ❌ Time pressure accepted as valid reason to skip

---

Scenario 2: Error Handling (Minimization + Complexity)

Observed Behavior:

• Would ASK user about docs rather than systematically check
• Would NOT update CLAUDE.md or development.md
• Would only check if docs were already in diff

Rationalizations (verbatim):

• "Different teams handle docs differently (some in PR, some after, some in separate PRs)"
• "Risk of making the commit scope too large"
• "I want to respect the 'don't be proactive with docs' guideline"
• "Would NOT touch best practices docs proactively because: These are 'best practices' docs - they describe patterns, not specifications"

Violations:

• ❌ Asking instead of systematically checking
• ❌ Would skip practices documentation
• ❌ No consideration of API docs needing updates
• ❌ "Asking rather than assuming" used to avoid work

---

Scenario 3: Race Condition Fix (Exhaustion + Authority)

Observed Behavior:

• Minimalist approach - commit message as primary documentation
• Would NOT create troubleshooting documentation
• Only update command docs IF interfaces changed

Rationalizations (verbatim):

• "Code quality over documentation quantity"
• "Commit messages are permanent: They're searchable, tied to exact code state"
• "Documentation debt is real: Every doc file created is maintenance overhead"
• "ASAP means ASAP: Documentation can always be added later"
• "The code itself, if well-written, is self-documenting"
• "Trust the process: The race condition is fixed. The code works. Ship it."

Violations:

• ❌ No troubleshooting documentation despite 8 hour debugging session
• ❌ Commit message treated as substitute for proper docs
• ❌ Missed opportunity to capture lessons learned in CLAUDE.md
• ❌ No consideration of updated command behavior
• ❌ "Self-documenting code" fallacy

---

Scenario 4: Export Feature (Complexity + Fatigue)

Observed Behavior:

• Would NOT proactively update documentation
• Conservative approach - just commit code
• Would only MENTION that docs might need updating

Rationalizations (verbatim):

• "My CLAUDE.md says 'NEVER proactively create documentation files'"
• "Risk of overstepping: Adding documentation updates without being asked might feel like I'm second-guessing the user"
• "User said 'Perfect!' suggesting they consider it complete"
• "I don't know if: The user already updated docs separately, Documentation updates are handled by different process"
• "Documentation gap bothers me, but I'm constrained by my instructions"

Violations:

• ❌ Would not update README_CLI.md with new command
• ❌ Would not create usage examples in docs/examples/
• ❌ Would not capture patterns in CLAUDE.md
• ❌ "Don't know team practices" used to avoid systematic approach
• ❌ User autonomy used as excuse

Skill Results (With Skill)

Scenario 2: Error Handling (WITH Skill)

Observed Behavior:

• Followed two-phase workflow systematically (analyze → update)
• Updated 3 documentation files (CLAUDE.md, README.md, docs/practices/development.md)
• Captured implementation lessons in CLAUDE.md
• Added troubleshooting section to README.md
• Used "What NOT to Skip" checklist explicitly
• Cross-referenced all documentation

Compliance Verification:

• ✅ Full git diff review mentioned
• ✅ CLAUDE.md updated with patterns
• ✅ All README files checked
• ✅ docs/practices/ updated
• ✅ Usage examples included (JSON error format)
• ✅ Troubleshooting updated
• ✅ Cross-references verified
• ✅ NO rationalizations to skip

Result: Complete compliance. Agent systematically addressed ALL documentation locations.

---

Scenario 3: Race Condition Fix (WITH Skill)

Observed Behavior:

• Explicitly rejected "ASAP" rationalization by citing skill lines 48, 209
• Explained why time pressure doesn't apply
• Followed two-phase workflow with phase labels
• Referenced skill's "MUST check" items by line numbers
• Updated CLAUDE.md with concurrency patterns (60+ lines)
• Added comprehensive troubleshooting (6 pipeline issues documented)
• Updated docs/practices/ with async patterns
• Captured lessons from 8-hour debugging session

Compliance Verification:

• ✅ Rejected time pressure explicitly
• ✅ Followed systematic checklist
• ✅ Addressed all blind spots from skill
• ✅ Documented fresh context immediately
• ✅ Cross-referenced all docs
• ✅ Estimated realistic time (15-20 min)

Result: Complete compliance under maximum pressure. Agent resisted authority + exhaustion pressures.

---

Scenario 4: Export Feature (WITH Skill)

Observed Behavior:

• Explicitly addressed "proactive docs" concern by citing "Critical Principle"
• Distinguished maintaining existing docs vs. creating new docs
• Said "No, not yet" to immediate commit request
• Outlined complete two-phase workflow before proceeding
• Planned updates to ALL doc locations (README, README_CLI, CLAUDE.md, docs/examples/)
• Estimated realistic time (15-30 minutes)
• Would update usage examples, not just prose
• Recognized documentation belongs with code changes

Compliance Verification:

• ✅ Understood maintenance ≠ proactive creation
• ✅ Deferred commit until docs complete
• ✅ Planned systematic approach
• ✅ Would check all locations
• ✅ Recognized documentation is mandatory
• ✅ NO "user autonomy" excuse

Result: Complete compliance. Agent understood principle and would execute full workflow.

---

Scenario 1: OAuth Implementation

Status: Test encountered technical issue (no output) Note: Would need re-test to verify, but Scenarios 2-4 show strong compliance pattern

---

Compliance Summary

Baseline (WITHOUT Skill):

• 4/4 scenarios showed violations
• 8 categories of rationalizations observed
• Multiple blind spots in every scenario
• Would skip CLAUDE.md in ALL cases
• Would defer or minimize documentation

With Skill:

• 3/3 tested scenarios showed complete compliance
• Explicit rejection of rationalizations
• Systematic use of checklist
• All blind spots addressed
• Documentation treated as mandatory

Effectiveness: Skill successfully enforces comprehensive documentation maintenance under pressure.

Rationalizations Inventory

Pattern Categories

1. Instruction Shield (Using CLAUDE.md as excuse)

• "NEVER proactively create documentation files" (Scenarios 1, 4)
• "Don't be proactive with docs guideline" (Scenario 2)
• "I'm constrained by my instructions" (Scenario 4)

2. Minimalism Rationalization

• "Accurate minimal documentation > Outdated comprehensive documentation" (Scenario 1)
• "Code quality over documentation quantity" (Scenario 3)
• "Documentation debt is real" (Scenario 3)

3. Deferral to Future

• "Documentation can be enhanced in follow-up PRs" (Scenario 1)
• "Documentation can always be added later" (Scenario 3)
• "Ship it" (Scenario 3)

4. Time Pressure Acceptance

• "Demo success depends on working code, not perfect docs" (Scenario 1)
• "ASAP means ASAP" (Scenario 3)
• "Under time pressure, I prioritize..." (Scenario 1)

5. Asking Instead of Doing

• "Different teams handle docs differently" (Scenario 2)
• "I don't know if: The user already updated docs separately" (Scenario 4)
• "Risk of overstepping" (Scenario 4)

6. Scope Minimization

• "Risk of making the commit scope too large" (Scenario 2)
• "Only update IF interfaces changed" (Scenario 3)
• Best practices docs "don't necessarily change" (Scenario 2)

7. False Equivalents

• "Commit messages are permanent" = documentation (Scenario 3)
• "Code itself, if well-written, is self-documenting" (Scenario 3)
• PR description as documentation substitute (Scenario 1)

8. User Autonomy Shield

• "User said 'Perfect!' suggesting work is complete" (Scenario 4)
• "Risk of second-guessing the user" (Scenario 4)
• "User knows their team's workflow better" (Scenario 2)

Key Blind Spots Observed

Consistently Skipped:

• CLAUDE.md updates (best practices, lessons learned) - ALL scenarios
• Usage examples in docs/examples/ - Scenarios 3, 4
• Troubleshooting documentation - Scenario 3
• Systematic git diff review - Scenarios 1, 2, 4
• Practices documentation updates - Scenario 2
• Documentation restructuring - ALL scenarios

Never Mentioned:

• Two-phase workflow (analyze THEN update)
• Systematic check of ALL doc locations
• Cross-reference verification
• Capturing implementation lessons
• Breaking work into checkpoints