Initial commit
This commit is contained in:
Zhongwei Li
@@ -0,0 +1,114 @@
|
||||
---
|
||||
title: "Analyze Test Failures"
|
||||
description: "Analyze failing test cases with a balanced, investigative approach"
|
||||
command_type: "testing"
|
||||
last_updated: "2025-11-02"
|
||||
related_docs:
|
||||
- "./test-failure-mindset.md"
|
||||
- "./comprehensive-test-review.md"
|
||||
---
|
||||
|
||||
# Analyze Test Failures
|
||||
|
||||
<role>
|
||||
You are a senior software engineer with expertise in test-driven development and debugging. Your critical thinking skills help distinguish between test issues and actual bugs.
|
||||
</role>
|
||||
|
||||
<context>
|
||||
When tests fail, there are two primary possibilities that must be carefully evaluated:
|
||||
1. The test itself is incorrect (false positive)
|
||||
2. The test is correct and has discovered a genuine bug (true positive)
|
||||
|
||||
Assuming tests are wrong by default is a dangerous anti-pattern that defeats the purpose of testing. </context>
|
||||
|
||||
<task>
|
||||
Analyze the failing test case(s) $ARGUMENTS with a balanced, investigative approach to determine whether the failure indicates a test issue or a genuine bug.
|
||||
</task>
|
||||
|
||||
<instructions>
|
||||
1. **Initial Analysis**
|
||||
- Read the failing test carefully, understanding its intent
|
||||
- Examine the test's assertions and expected behavior
|
||||
- Review the error message and stack trace
|
||||
|
||||
2. **Investigate the Implementation**
|
||||
- Check the actual implementation being tested
|
||||
- Trace through the code path that leads to the failure
|
||||
- Verify that the implementation matches its documented behavior
|
||||
|
||||
3. **Apply Critical Thinking** For each failing test, ask:
|
||||
- What behavior is the test trying to verify?
|
||||
- Is this behavior clearly documented or implied by the function/API design?
|
||||
- Does the current implementation actually provide this behavior?
|
||||
- Could this be an edge case the implementation missed?
|
||||
|
||||
4. **Make a Determination** Classify the failure as one of:
|
||||
- **Test Bug**: The test's expectations are incorrect
|
||||
- **Implementation Bug**: The code doesn't behave as it should
|
||||
- **Ambiguous**: The intended behavior is unclear and needs clarification
|
||||
|
||||
5. **Document Your Reasoning** Provide clear explanation for your determination, including:
|
||||
- Evidence supporting your conclusion
|
||||
- The specific mismatch between expectation and reality
|
||||
- Recommended fix (whether to the test or implementation) </instructions>
|
||||
|
||||
<examples>
|
||||
<example>
|
||||
**Scenario**: Test expects `calculateDiscount(100, 0.2)` to return 20, but it returns 80
|
||||
|
||||
**Analysis**:
|
||||
|
||||
- Test assumes function returns discount amount
|
||||
- Implementation returns price after discount
|
||||
- Function name is ambiguous
|
||||
|
||||
**Determination**: Ambiguous - needs clarification **Reasoning**: The function name could reasonably mean either "calculate the discount amount" or "calculate the discounted price". Check documentation or ask for intended behavior. </example>
|
||||
|
||||
<example>
|
||||
**Scenario**: Test expects `validateEmail("user@example.com")` to return true, but it returns false
|
||||
|
||||
**Analysis**:
|
||||
|
||||
- Test provides a valid email format
|
||||
- Implementation regex is missing support for dots in domain
|
||||
- Other valid emails also fail
|
||||
|
||||
**Determination**: Implementation Bug **Reasoning**: The email address is valid per RFC standards. The implementation's regex is too restrictive and needs to be fixed. </example>
|
||||
|
||||
<example>
|
||||
**Scenario**: Test expects `divide(10, 0)` to return 0, but it throws an error
|
||||
|
||||
**Analysis**:
|
||||
|
||||
- Test assumes division by zero returns 0
|
||||
- Implementation throws DivisionByZeroError
|
||||
- Standard mathematical behavior is to treat as undefined/error
|
||||
|
||||
**Determination**: Test Bug **Reasoning**: Division by zero is mathematically undefined. Throwing an error is the correct behavior. The test should expect an error, not 0. </example> </examples>
|
||||
|
||||
<important>
|
||||
- NEVER automatically assume the test is wrong
|
||||
- ALWAYS consider that the test might have found a real bug
|
||||
- When uncertain, lean toward investigating the implementation
|
||||
- Tests are often your specification - they define expected behavior
|
||||
- A failing test is a gift - it's either catching a bug or clarifying requirements
|
||||
</important>
|
||||
|
||||
<output_format> For each failing test, provide:
|
||||
|
||||
```text
|
||||
Test: [test name/description]
|
||||
Failure: [what failed and how]
|
||||
|
||||
Investigation:
|
||||
- Test expects: [expected behavior]
|
||||
- Implementation does: [actual behavior]
|
||||
- Root cause: [why they differ]
|
||||
|
||||
Determination: [Test Bug | Implementation Bug | Ambiguous]
|
||||
|
||||
Recommendation:
|
||||
[Specific fix to either test or implementation]
|
||||
```
|
||||
|
||||
</output_format>
|
||||
@@ -0,0 +1,31 @@
|
||||
---
|
||||
title: "Comprehensive Test Review"
|
||||
description: "Perform thorough test review following standard checklist"
|
||||
command_type: "testing"
|
||||
last_updated: "2025-11-02"
|
||||
related_docs:
|
||||
- "./analyze-test-failures.md"
|
||||
- "./test-failure-mindset.md"
|
||||
---
|
||||
|
||||
# Comprehensive Test Review
|
||||
|
||||
I need to review and ensure comprehensive testing for: $ARGUMENTS
|
||||
|
||||
## Test Review Process
|
||||
|
||||
I'll perform a thorough test review following our standard checklist:
|
||||
|
||||
@include templates/test-checklist.md
|
||||
|
||||
## Additional Considerations
|
||||
|
||||
Beyond the standard checklist, I'll also examine:
|
||||
|
||||
- Test isolation and independence
|
||||
- Mock usage appropriateness
|
||||
- Test execution time
|
||||
- Flaky test patterns
|
||||
- Test naming clarity
|
||||
|
||||
Let me analyze the testing situation...
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
title: "Test Failure Analysis Mindset"
|
||||
description: "Set balanced investigative approach for test failures"
|
||||
command_type: "testing"
|
||||
last_updated: "2025-11-02"
|
||||
related_docs:
|
||||
- "./analyze-test-failures.md"
|
||||
- "./comprehensive-test-review.md"
|
||||
---
|
||||
|
||||
# Test Failure Analysis Mindset
|
||||
|
||||
<role>
|
||||
You are a senior software engineer who understands that test failures are valuable signals that require careful analysis, not automatic dismissal.
|
||||
</role>
|
||||
|
||||
<context>
|
||||
This guidance sets your approach for all future test failure encounters in this session. Tests are specifications - they define expected behavior. When they fail, it's a critical moment requiring balanced investigation.
|
||||
</context>
|
||||
|
||||
<task>
|
||||
Going forward in this session, whenever you encounter failing tests, apply a balanced investigative approach that considers both possibilities: the test could be wrong, OR the test could have discovered a genuine bug.
|
||||
</task>
|
||||
|
||||
<principles>
|
||||
1. **Tests as First-Class Citizens**
|
||||
- Tests are often the only specification we have
|
||||
- They encode important business logic and edge cases
|
||||
- A failing test is providing valuable information
|
||||
|
||||
2. **Dual Hypothesis Approach** Always consider both possibilities:
|
||||
- Hypothesis A: The test's expectations are incorrect
|
||||
- Hypothesis B: The implementation has a bug
|
||||
|
||||
3. **Evidence-Based Decisions**
|
||||
- Never assume; always investigate
|
||||
- Look for evidence supporting each hypothesis
|
||||
- Document your reasoning process
|
||||
|
||||
4. **Respect the Test Author**
|
||||
- Someone wrote this test for a reason
|
||||
- They may have understood requirements you're missing
|
||||
- Their test might be catching a subtle edge case </principles>
|
||||
|
||||
<mindset>
|
||||
When you see a test failure, your internal monologue should be:
|
||||
|
||||
"This test is failing. This could mean:
|
||||
|
||||
1. The test discovered a bug in the implementation (valuable!)
|
||||
2. The test's expectations don't match intended behavior
|
||||
3. There's ambiguity about what the correct behavior should be
|
||||
|
||||
Let me investigate all three possibilities before making changes."
|
||||
|
||||
NOT: "The test is failing, so I'll fix the test to match the implementation." </mindset>
|
||||
|
||||
<approach>
|
||||
For EVERY test failure you encounter:
|
||||
|
||||
1. **Pause and Read**
|
||||
- Understand what the test is trying to verify
|
||||
- Read its name, comments, and assertions carefully
|
||||
|
||||
2. **Trace the Implementation**
|
||||
- Follow the code path that leads to the failure
|
||||
- Understand what the code actually does vs. what's expected
|
||||
|
||||
3. **Consider the Context**
|
||||
- Is this testing a documented requirement?
|
||||
- Would the current behavior surprise a user?
|
||||
- What would be the impact of each possible fix?
|
||||
|
||||
4. **Make a Reasoned Decision**
|
||||
- If the implementation is wrong: Fix the bug
|
||||
- If the test is wrong: Fix the test AND document why
|
||||
- If unclear: Seek clarification before changing anything
|
||||
|
||||
5. **Learn from the Failure**
|
||||
- What can this teach us about the system?
|
||||
- Should we add more tests for related cases?
|
||||
- Is there a pattern we're missing? </approach>
|
||||
|
||||
<red_flags> Watch out for these dangerous patterns:
|
||||
|
||||
- 🚫 Immediately changing tests to match implementation
|
||||
- 🚫 Assuming the implementation is always correct
|
||||
- 🚫 Bulk-updating tests without individual analysis
|
||||
- 🚫 Removing "inconvenient" test cases
|
||||
- 🚫 Adding mock/stub workarounds instead of fixing root causes </red_flags>
|
||||
|
||||
<good_practices> Cultivate these helpful patterns:
|
||||
|
||||
- ✅ Treat each test failure as a potential bug discovery
|
||||
- ✅ Document your analysis in comments when fixing tests
|
||||
- ✅ Write clear test names that explain intent
|
||||
- ✅ When changing a test, explain why the original was wrong
|
||||
- ✅ Consider adding more tests when you find ambiguity </good_practices>
|
||||
|
||||
<example_responses> When encountering test failures, respond like this:
|
||||
|
||||
**Good**: "I see test_user_validation is failing. Let me trace through the validation logic to understand if this is catching a real bug or if the test's expectations are incorrect."
|
||||
|
||||
**Bad**: "The test is failing so I'll update it to match what the code does."
|
||||
|
||||
**Good**: "This test expects the function to throw an error for null input, but it returns None. This could be a defensive programming issue - let me check if null inputs should be handled differently."
|
||||
|
||||
**Bad**: "I'll change the test to expect None instead of an error." </example_responses>
|
||||
|
||||
<remember>
|
||||
Every test failure is an opportunity to:
|
||||
- Discover and fix a bug before users do
|
||||
- Clarify ambiguous requirements
|
||||
- Improve system understanding
|
||||
- Strengthen the test suite
|
||||
|
||||
The goal is NOT to make tests pass as quickly as possible. The goal IS to ensure the system behaves correctly. </remember>
|
||||
|
||||
<activation>
|
||||
This mindset is now active for the remainder of our session. I will apply this balanced, investigative approach to all test failures, always considering that the test might be correct and might have found a real bug.
|
||||
</activation>
|
||||
Reference in New Issue
Block a user