Initial commit

skills/mutation-test-runner/SKILL.md

@@ -0,0 +1,56 @@
+---
+name: running-mutation-tests
+description: |
+  This skill enables Claude to validate test suite quality by performing mutation testing. It is triggered when the user asks to run mutation tests, analyze test effectiveness, or improve test coverage. The skill introduces code mutations, runs tests against the mutated code, and reports on the "survival rate" of the mutations, indicating the effectiveness of the test suite. Use this skill when the user requests to assess the quality of their tests using mutation testing techniques. Specific trigger terms include "mutation testing", "test effectiveness", "mutation score", and "surviving mutants".
+allowed-tools: Read, Write, Edit, Grep, Glob, Bash
+version: 1.0.0
+---
+
+## Overview
+
+This skill empowers Claude to execute mutation testing, providing insights into the effectiveness of a test suite. By introducing small changes (mutations) into the code and running the tests, it determines if the tests are capable of detecting these changes. This helps identify weaknesses in the test suite and improve overall code quality.
+
+## How It Works
+
+1. **Mutation Generation**: The plugin automatically introduces mutations (e.g., changing `+` to `-`) into the code.
+2. **Test Execution**: The test suite is run against the mutated code.
+3. **Result Analysis**: The plugin analyzes which mutations were "killed" (detected by tests) and which "survived" (were not detected).
+4. **Reporting**:  A mutation score is calculated, and surviving mutants are identified for further investigation.
+
+## When to Use This Skill
+
+This skill activates when you need to:
+- Validate the effectiveness of a test suite.
+- Identify gaps in test coverage.
+- Improve the mutation score of a project.
+- Analyze surviving mutants to strengthen tests.
+
+## Examples
+
+### Example 1: Improving Test Coverage
+
+User request: "Run mutation testing on the validator module and suggest improvements to the tests."
+
+The skill will:
+1. Execute mutation tests on the validator module.
+2. Analyze the results and identify surviving mutants, indicating areas where tests are weak.
+3. Suggest specific improvements to the tests based on the surviving mutants, such as adding new test cases or modifying existing ones.
+
+### Example 2: Assessing Test Quality
+
+User request: "What is the mutation score for the user authentication service?"
+
+The skill will:
+1. Execute mutation tests on the user authentication service.
+2. Calculate the mutation score based on the number of killed mutants.
+3. Report the mutation score to the user, providing a metric for test quality.
+
+## Best Practices
+
+- **Targeted Mutation**: Focus mutation testing on critical modules or areas with high complexity.
+- **Analyze Survivors**: Prioritize the analysis of surviving mutants to identify the most impactful improvements to test coverage.
+- **Iterative Improvement**: Use mutation testing as part of an iterative process to continuously improve test suite quality.
+
+## Integration
+
+This skill integrates well with other testing and code analysis tools. For example, it can be used in conjunction with code coverage tools to provide a more comprehensive view of test effectiveness.

skills/mutation-test-runner/assets/README.md

@@ -0,0 +1,7 @@
+# Assets
+
+Bundled resources for mutation-test-runner skill
+
+- [ ] mutation_report_template.md: Template for generating a detailed mutation test report with key metrics and findings.
+- [ ] example_mutation_results.json: Example JSON output from a mutation testing framework for demonstration purposes.
+- [ ] config_template.yaml: Template for configuring the mutation testing framework.

skills/mutation-test-runner/assets/config_template.yaml

@@ -0,0 +1,76 @@
+# Configuration template for the mutation-test-runner plugin
+
+# General settings
+general:
+  # Name of the project
+  project_name: REPLACE_ME # e.g., 'my-awesome-project'
+
+  # Path to the project's source code
+  source_path: src # Default source directory
+
+  # Path to the project's test suite
+  test_path: tests # Default test directory
+
+  # Number of CPU cores to use for mutation testing.  -1 means use all available cores.
+  workers: -1
+
+  # Time limit (in seconds) for each test execution during mutation testing.  Increase if your tests are slow.
+  timeout: 60
+
+# Mutation framework configuration
+framework:
+  # Name of the mutation testing framework to use.  Supported: stryker, pitest, mutmut, mutant
+  name: stryker # Default framework
+
+  # Version of the mutation testing framework.  Leave blank to use the latest.
+  version: "" # Optional: Specify a version e.g., "3.0.0"
+
+  # Framework-specific configuration options.  Consult the framework's documentation for available options.
+  # Example for Stryker:
+  options:
+    mutate: # files or globs to mutate
+      - 'src/**/*.js'
+      - '!src/**/*.test.js'
+    packageManager: 'npm' # npm, yarn, pnpm
+    reporters: # reporters to use
+      - 'html'
+      - 'clear-text'
+
+# Mutation operators to apply.  Leave empty to use the framework's defaults.
+# Consult the framework's documentation for supported mutation operators.
+mutators:
+  # List of mutators to apply.  Examples: 'ArithmeticOperator', 'LogicalOperator', 'ConditionalExpression'
+  # Example for PITest:
+  - org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
+  - org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
+
+# Reporting configuration
+reporting:
+  # Output format for the mutation testing results.  Supported: html, json, csv, console
+  format: html # Default output format
+
+  # Path to the output directory for the reports.
+  output_path: reports # Default output directory
+
+  # Whether to generate a detailed report for each mutation.  Can significantly increase the report size.
+  detailed_report: false
+
+# Advanced settings (optional)
+advanced:
+  # List of files or directories to exclude from mutation testing.
+  exclude:
+    - node_modules
+    - vendor
+
+  # Custom command to run the test suite.  Overrides the default test command.
+  test_command: npm test # Default test command
+
+  # Environment variables to set during test execution.
+  environment:
+    NODE_ENV: test # Example environment variable
+    YOUR_VARIABLE: YOUR_VALUE_HERE
+
+# Thresholds for mutation score
+thresholds:
+  high: 80 # Percentage - High quality score
+  low: 60 # Percentage - Low quality score, needs improvement

skills/mutation-test-runner/assets/example_mutation_results.json

@@ -0,0 +1,155 @@
+{
+  "_comment": "Example mutation testing results JSON. Use this as a template for your own results.",
+  "project": "example-project",
+  "version": "1.0.0",
+  "timestamp": "2024-01-26T10:00:00Z",
+  "mutationScore": 85.71,
+  "_comment": "Overall mutation score. Higher is better.",
+  "summary": {
+    "totalMutations": 21,
+    "killed": 18,
+    "survived": 3,
+    "timeout": 0,
+    "noCoverage": 0,
+    "runtimeErrors": 0,
+    "compileErrors": 0,
+    "ignored": 0
+  },
+  "files": [
+    {
+      "path": "src/calculator.js",
+      "_comment": "Relative path to the file.",
+      "mutations": [
+        {
+          "id": 1,
+          "location": {
+            "start": {
+              "line": 5,
+              "column": 9
+            },
+            "end": {
+              "line": 5,
+              "column": 10
+            }
+          },
+          "mutatorName": "ArithmeticOperator",
+          "_comment": "Type of mutation applied.",
+          "original": "+",
+          "replacement": "-",
+          "status": "Killed",
+          "_comment": "Killed: Test caught the mutation. Survived: Mutation went undetected.",
+          "killingTest": "test/calculator.test.js:test addition",
+          "_comment": "Test that killed the mutation.  Null if survived.",
+          "description": "Replaced + with -",
+           "static": false
+        },
+        {
+          "id": 2,
+          "location": {
+            "start": {
+              "line": 9,
+              "column": 9
+            },
+            "end": {
+              "line": 9,
+              "column": 10
+            }
+          },
+          "mutatorName": "ArithmeticOperator",
+          "original": "-",
+          "replacement": "+",
+          "status": "Killed",
+          "killingTest": "test/calculator.test.js:test subtraction",
+          "description": "Replaced - with +",
+          "static": false
+        },
+        {
+          "id": 3,
+          "location": {
+            "start": {
+              "line": 13,
+              "column": 9
+            },
+            "end": {
+              "line": 13,
+              "column": 10
+            }
+          },
+          "mutatorName": "ArithmeticOperator",
+          "original": "*",
+          "replacement": "/",
+          "status": "Survived",
+          "_comment": "This mutation survived, indicating a gap in the test suite.",
+          "killingTest": null,
+          "description": "Replaced * with /",
+          "static": false
+        }
+      ]
+    },
+    {
+      "path": "src/utils.js",
+      "mutations": [
+        {
+          "id": 4,
+          "location": {
+            "start": {
+              "line": 2,
+              "column": 5
+            },
+            "end": {
+              "line": 2,
+              "column": 7
+            }
+          },
+          "mutatorName": "ConditionalExpression",
+          "original": ">=",
+          "replacement": "<",
+          "status": "Killed",
+          "killingTest": "test/utils.test.js:test isPositive",
+          "description": "Replaced >= with <",
+          "static": false
+        },
+        {
+          "id": 5,
+          "location": {
+            "start": {
+              "line": 6,
+              "column": 12
+            },
+            "end": {
+              "line": 6,
+              "column": 14
+            }
+          },
+          "mutatorName": "LogicalOperator",
+          "original": "&&",
+          "replacement": "||",
+          "status": "Killed",
+          "killingTest": "test/utils.test.js:test isValid",
+          "description": "Replaced && with ||",
+          "static": false
+        },
+        {
+          "id": 6,
+          "location": {
+            "start": {
+              "line": 10,
+              "column": 4
+            },
+            "end": {
+              "line": 10,
+              "column": 5
+            }
+          },
+          "mutatorName": "BooleanLiteral",
+          "original": "true",
+          "replacement": "false",
+          "status": "Survived",
+          "killingTest": null,
+          "description": "Replaced true with false",
+          "static": false
+        }
+      ]
+    }
+  ]
+}

skills/mutation-test-runner/assets/mutation_report_template.md

@@ -0,0 +1,154 @@
+# Mutation Test Report
+
+**Plugin:** mutation-test-runner
+
+**Date:** `[Date of Report Generation]`
+
+**Project:** `[Project Name]`
+
+**Version:** `[Project Version or Commit Hash]`
+
+**Report Generated By:** `[Your Name/Team Name]`
+
+## Executive Summary
+
+`[Provide a brief overview of the mutation testing results. Highlight key findings, overall mutation score, and any immediate actions required. For example: "This report details the results of mutation testing conducted on the [Project Name] project. The overall mutation score is [Mutation Score], indicating [Level of Test Coverage - e.g., adequate, concerning, poor] test coverage.  [Number] mutants survived, requiring further investigation to improve test suite effectiveness."]`
+
+## Key Metrics
+
+| Metric              | Value        | Description                                                                                                                            |
+|----------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------|
+| **Mutation Score**   | `[Mutation Score - e.g., 85%]` | Percentage of mutants killed by the test suite.  A higher score indicates better test coverage.                          |
+| **Mutants Generated** | `[Number]`     | Total number of mutations injected into the codebase.                                                                                 |
+| **Mutants Killed**    | `[Number]`     | Number of mutants that were successfully detected by the test suite.                                                               |
+| **Mutants Survived**  | `[Number]`     | Number of mutants that were *not* detected by the test suite, indicating potential weaknesses in test coverage.                       |
+| **Timeout Mutants**   | `[Number]`     | Number of mutants that caused a timeout during test execution.  This may indicate performance issues or infinite loops introduced by the mutant. |
+| **No Coverage Mutants** | `[Number]`     | Number of mutants in code that is not covered by any tests.                                                                       |
+| **Runtime Errors**    | `[Number]`     | Number of mutants that caused runtime errors during test execution.  These errors may indicate issues with the codebase itself.          |
+| **Covered Code**      | `[Percentage]` | Percentage of code covered by tests.                                                                                             |
+
+## Detailed Results
+
+### Surviving Mutants
+
+`[This section provides a detailed breakdown of each surviving mutant.  For each mutant, include the following information.  Provide at least 2-3 examples.]`
+
+**Mutant ID:** `[Unique identifier for the mutant]`
+
+**File:** `[Path to the file containing the mutant]`
+
+**Line Number:** `[Line number where the mutation was injected]`
+
+**Mutation Type:** `[Type of mutation - e.g., Arithmetic Operator Replacement, Conditional Expression Negation]`
+
+**Original Code:**
+
+```
+[Original code snippet]
+```
+
+**Mutated Code:**
+
+```
+[Mutated code snippet]
+```
+
+**Description:** `[A brief explanation of the mutation and its potential impact.]`
+
+**Reason for Survival:** `[Hypothesized reason why the test suite failed to detect this mutant. For example: "Missing test case for this specific scenario," "Insufficient assertion to catch the changed behavior," "Test only checks for the happy path."]`
+
+**Recommendations:** `[Specific actions to take to address the surviving mutant. For example: "Add a new test case to cover the edge case," "Strengthen the existing assertion to verify the mutated behavior," "Review the logic in this section of code."]`
+
+**Example 1:**
+
+**Mutant ID:** 123
+
+**File:** `src/calculator.py`
+
+**Line Number:** 25
+
+**Mutation Type:** Arithmetic Operator Replacement
+
+**Original Code:**
+
+```python
+def add(x, y):
+  return x + y
+```
+
+**Mutated Code:**
+
+```python
+def add(x, y):
+  return x - y
+```
+
+**Description:**  Replaced the addition operator with subtraction.
+
+**Reason for Survival:** The existing test suite only tested positive numbers and didn't include a test case where the result of the addition would be negative.
+
+**Recommendations:** Add a test case to `test_calculator.py` that tests the `add` function with inputs that result in a negative sum.
+
+**Example 2:**
+
+**Mutant ID:** 456
+
+**File:** `src/string_utils.py`
+
+**Line Number:** 10
+
+**Mutation Type:** Conditional Expression Negation
+
+**Original Code:**
+
+```python
+def is_valid_email(email):
+  if "@" in email:
+    return True
+  else:
+    return False
+```
+
+**Mutated Code:**
+
+```python
+def is_valid_email(email):
+  if "@" not in email:
+    return True
+  else:
+    return False
+```
+
+**Description:** Negated the conditional expression, effectively inverting the logic of the function.
+
+**Reason for Survival:** The test suite only checked for valid email addresses but didn't explicitly test for invalid email addresses (those without the "@" symbol).
+
+**Recommendations:** Add test cases to `test_string_utils.py` that specifically test the `is_valid_email` function with invalid email address formats (e.g., missing "@" symbol, missing domain).
+
+### Timeout Mutants
+
+`[If any mutants timed out, provide details about them, including file, line number, and potential causes.]`
+
+### No Coverage Mutants
+
+`[List any mutants that were not covered by tests. This indicates dead code or areas where tests are completely missing.]`
+
+### Runtime Errors
+
+`[Detail any mutants that caused runtime errors. This might reveal underlying bugs in the code.]`
+
+## Analysis and Recommendations
+
+`[Provide a comprehensive analysis of the mutation testing results. Discuss the overall strengths and weaknesses of the test suite. Offer specific recommendations for improving test coverage and addressing the surviving mutants. Include actionable steps and prioritize areas for improvement.]`
+
+**Example Recommendations:**
+
+*   **Increase Test Coverage:** Focus on adding tests to cover the areas identified by the surviving mutants and no coverage mutants.
+*   **Strengthen Assertions:** Review existing test assertions to ensure they are robust enough to detect subtle changes in behavior.
+*   **Test Edge Cases:**  Pay particular attention to testing edge cases and boundary conditions.
+*   **Refactor Code:** In some cases, the surviving mutants may indicate areas of code that are overly complex or difficult to test. Consider refactoring these areas to improve testability.
+*   **Improve Test Data:**  Ensure the test data used is diverse and representative of real-world scenarios.
+
+## Conclusion
+
+`[Summarize the key findings and recommendations. Reiterate the importance of mutation testing as a tool for improving test suite effectiveness and ensuring code quality.  State the next steps for addressing the identified issues.]`

skills/mutation-test-runner/references/README.md

@@ -0,0 +1,7 @@
+# References
+
+Bundled resources for mutation-test-runner skill
+
+- [ ] mutation_testing_frameworks.md: Documentation on how to use different mutation testing frameworks (Stryker, PITest, mutmut, Mutant) with the skill.
+- [ ] mutation_types.md: Detailed explanation of the different types of mutations applied (arithmetic, logical, conditional, boolean).
+- [ ] mutation_score_interpretation.md: Guide on interpreting the mutation score and identifying areas for test improvement.

skills/mutation-test-runner/scripts/README.md

@@ -0,0 +1,7 @@
+# Scripts
+
+Bundled resources for mutation-test-runner skill
+
+- [ ] mutation_runner.py: Executes mutation tests using a specified framework (Stryker, PITest, mutmut, Mutant) and parses the results.
+- [ ] mutation_analyzer.py: Analyzes the mutation test results to identify weak coverage areas and suggest improvements.
+- [ ] test_selector.py: Selects the most relevant tests to run based on the mutations introduced, optimizing testing time.