Initial commit

2025-11-29 18:16:56 +08:00
commit 8a3d331e04
61 changed files with 11808 additions and 0 deletions
--- a/skills/skill-isolation-tester/test-templates/README.md
+++ b/skills/skill-isolation-tester/test-templates/README.md
@@ -0,0 +1,392 @@
+# Skill Test Templates
+
+Production-ready test templates for validating Claude Code skills in isolated environments.
+
+## Overview
+
+These templates provide standardized testing workflows for different skill types. Each template includes:
+- Pre-flight environment validation
+- Before/after snapshots for comparison
+- Comprehensive safety and security checks
+- Detailed reporting with pass/fail criteria
+- Automatic cleanup on exit (success or failure)
+
+## CI/CD Integration with JSON Output
+
+All test templates support JSON output for integration with CI/CD pipelines. The JSON reporter generates:
+- **Structured JSON** - Machine-readable test results
+- **JUnit XML** - Compatible with Jenkins, GitLab CI, GitHub Actions
+- **Markdown Summary** - Human-readable reports for GitHub Actions
+
+**Enable JSON output:**
+```bash
+export JSON_ENABLED=true
+./test-templates/docker-skill-test-json.sh my-skill
+```
+
+**Output files:**
+- `test-report.json` - Full structured test data
+- `test-report.junit.xml` - JUnit format for CI systems
+- `test-report.md` - Markdown summary
+
+**JSON Report Structure:**
+```json
+{
+  "test_name": "docker-skill-test",
+  "skill_name": "my-skill",
+  "timestamp": "2025-11-02T12:00:00Z",
+  "status": "passed",
+  "duration_seconds": 45,
+  "exit_code": 0,
+  "metrics": {
+    "containers_created": 2,
+    "images_created": 1,
+    "execution_duration_seconds": 12
+  },
+  "issues": [],
+  "recommendations": []
+}
+```
+
+**GitHub Actions Integration:**
+```yaml
+- name: Test Skill
+  run: |
+    export JSON_ENABLED=true
+    ./test-templates/docker-skill-test-json.sh my-skill
+
+- name: Upload Test Results
+  uses: actions/upload-artifact@v3
+  with:
+    name: test-results
+    path: /tmp/skill-test-*/test-report.*
+```
+
+See `lib/json-reporter.sh` for full API documentation.
+
+---
+
+## Available Templates
+
+### 1. Docker Skill Test (`docker-skill-test.sh`)
+
+**Use for skills that:**
+- Start or manage Docker containers
+- Build Docker images
+- Work with Docker volumes, networks, or compose files
+- Require Docker daemon access
+
+**Features:**
+- Tracks Docker resource creation (containers, images, volumes, networks)
+- Detects orphaned containers
+- Validates cleanup behavior
+- Resource limit enforcement
+
+**Usage:**
+```bash
+chmod +x test-templates/docker-skill-test.sh
+./test-templates/docker-skill-test.sh my-docker-skill
+```
+
+**Customization:**
+Edit the skill execution command on line ~178:
+```bash
+docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
+    cd /root/.claude/skills/$SKILL_NAME
+    ./skill.sh test-mode  # <-- Customize this
+"
+```
+
+---
+
+### 2. API Skill Test (`api-skill-test.sh`)
+
+**Use for skills that:**
+- Make HTTP/HTTPS requests to external APIs
+- Require API keys or authentication
+- Interact with web services
+- Need network access
+
+**Features:**
+- Network traffic monitoring
+- API call detection and counting
+- API key/secret leak detection
+- Rate limiting validation
+- HTTPS enforcement checking
+
+**Usage:**
+```bash
+chmod +x test-templates/api-skill-test.sh
+./test-templates/api-skill-test.sh my-api-skill
+```
+
+**Optional: Enable network capture:**
+```bash
+# Requires tcpdump and sudo
+sudo apt-get install tcpdump  # or brew install tcpdump
+./test-templates/api-skill-test.sh my-api-skill
+```
+
+---
+
+### 3. File Manipulation Skill Test (`file-manipulation-skill-test.sh`)
+
+**Use for skills that:**
+- Create, read, update, or delete files
+- Modify configuration files
+- Generate reports or artifacts
+- Perform filesystem operations
+
+**Features:**
+- Complete filesystem diff (added/removed/modified files)
+- File permission validation
+- Sensitive data scanning
+- Temp file cleanup verification
+- MD5 checksum comparison
+
+**Usage:**
+```bash
+chmod +x test-templates/file-manipulation-skill-test.sh
+./test-templates/file-manipulation-skill-test.sh my-file-skill
+```
+
+**Customization:**
+Add your own test files to the workspace (lines 54-70):
+```bash
+cat > "$TEST_DIR/test-workspace/your-file.txt" <<'EOF'
+Your test content here
+EOF
+```
+
+---
+
+### 4. Git Skill Test (`git-skill-test.sh`)
+
+**Use for skills that:**
+- Create commits, branches, or tags
+- Modify git history or configuration
+- Work with git worktrees
+- Interact with remote repositories
+
+**Features:**
+- Git state comparison (commits, branches, tags)
+- Working tree cleanliness validation
+- Force operation detection
+- History rewriting detection
+- Dangling commit detection
+
+**Usage:**
+```bash
+chmod +x test-templates/git-skill-test.sh
+./test-templates/git-skill-test.sh my-git-skill
+```
+
+**Customization:**
+Modify the test repository setup (lines 59-81) to match your skill's requirements.
+
+---
+
+## Common Usage Patterns
+
+### Basic Test Execution
+
+```bash
+# Run test for a specific skill
+./test-templates/docker-skill-test.sh my-skill-name
+
+# Keep container for debugging
+export SKILL_TEST_KEEP_CONTAINER="true"
+./test-templates/docker-skill-test.sh my-skill-name
+
+# Keep images after test
+export SKILL_TEST_REMOVE_IMAGES="false"
+./test-templates/docker-skill-test.sh my-skill-name
+```
+
+### Custom Resource Limits
+
+```bash
+# Set custom memory/CPU limits
+export SKILL_TEST_MEMORY_LIMIT="1g"
+export SKILL_TEST_CPU_LIMIT="2.0"
+./test-templates/docker-skill-test.sh my-skill-name
+```
+
+### Parallel Testing
+
+```bash
+# Test multiple skills in parallel
+for skill in skill1 skill2 skill3; do
+    ./test-templates/docker-skill-test.sh "$skill" &
+done
+wait
+echo "All tests complete!"
+```
+
+### CI/CD Integration
+
+```bash
+# Exit code 0 = pass, 1 = fail
+#!/bin/bash
+set -e
+
+SKILLS=(
+    "skill-creator"
+    "claude-code-otel-setup"
+    "playwright-e2e-automation"
+)
+
+for skill in "${SKILLS[@]}"; do
+    echo "Testing $skill..."
+    ./test-templates/docker-skill-test.sh "$skill" || {
+        echo "❌ $skill failed!"
+        exit 1
+    }
+done
+
+echo "✅ All skills passed!"
+```
+
+## Customizing Templates
+
+### Add Custom Validation
+
+Insert your own checks before the "Generate Test Report" section:
+
+```bash
+# ============================================================================
+# Custom Validation
+# ============================================================================
+
+echo ""
+echo "=== Running Custom Checks ==="
+
+# Your custom checks here
+docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
+    # Example: Check if specific file exists
+    test -f /workspace/expected-output.txt || {
+        echo 'ERROR: Expected output file not found'
+        exit 1
+    }
+"
+```
+
+### Modify Execution Command
+
+Each template has a skill execution section. Customize the command to match your skill's interface:
+
+```bash
+# Example: Run skill with arguments
+docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
+    cd /root/.claude/skills/$SKILL_NAME
+    ./skill.sh --mode=test --output=/workspace/results
+"
+
+# Example: Source skill as library
+docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
+    source /root/.claude/skills/$SKILL_NAME/lib.sh
+    run_skill_tests
+"
+```
+
+### Add Pre-Test Setup
+
+Insert setup steps after the "Build Test Environment" section:
+
+```bash
+# Install additional dependencies
+docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
+    apt-get update && apt-get install -y your-package
+"
+
+# Set environment variables
+docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
+    export SKILL_CONFIG_PATH=/etc/skill-config.json
+"
+```
+
+## Environment Variables
+
+All templates support these environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `SKILL_TEST_KEEP_CONTAINER` | `false` | Keep container after test for debugging |
+| `SKILL_TEST_REMOVE_IMAGES` | `true` | Remove test images after completion |
+| `SKILL_TEST_MEMORY_LIMIT` | `512m` | Container memory limit |
+| `SKILL_TEST_CPU_LIMIT` | `1.0` | Container CPU limit (cores) |
+| `SKILL_TEST_TEMP_DIR` | `/tmp/skill-test-*` | Temporary directory for test artifacts |
+
+## Exit Codes
+
+- `0` - Test passed (skill executed successfully)
+- `1` - Test failed (skill execution error or validation failure)
+- `>1` - Other errors (environment setup, Docker issues, etc.)
+
+## Troubleshooting
+
+### "Docker daemon not running"
+```bash
+# macOS
+open -a Docker
+
+# Linux
+sudo systemctl start docker
+```
+
+### "Permission denied" errors
+```bash
+# Add user to docker group
+sudo usermod -aG docker $USER
+newgrp docker
+```
+
+### Container hangs or never exits
+```bash
+# Set a timeout in your skill execution
+timeout 300 ./test-templates/docker-skill-test.sh my-skill
+```
+
+### Need to inspect failed test
+```bash
+# Keep container after failure
+export SKILL_TEST_KEEP_CONTAINER="true"
+./test-templates/docker-skill-test.sh my-skill
+
+# Inspect container
+docker start -ai <container-id>
+docker logs <container-id>
+```
+
+## Best Practices
+
+1. **Run tests before committing** - Catch environment-specific bugs early
+2. **Test in clean environment** - Don't rely on local configs or files
+3. **Validate cleanup** - Ensure skills don't leave orphaned resources
+4. **Check for secrets** - Never commit API keys or sensitive data
+5. **Document dependencies** - List all required packages and tools
+6. **Use resource limits** - Prevent runaway processes
+7. **Review diffs carefully** - Understand all file system changes
+
+## Contributing
+
+To add a new test template:
+
+1. Copy an existing template as a starting point
+2. Customize for your skill type
+3. Add comprehensive validation checks
+4. Update this README with usage documentation
+5. Test your template with at least 3 different skills
+
+## Related Documentation
+
+- `../lib/docker-helpers.sh` - Shared helper functions
+- `../modes/mode2-docker.md` - Docker isolation mode documentation
+- `../skill.md` - Main skill documentation
+
+## Support
+
+For issues or questions:
+- Check the skill logs: `docker logs <container-id>`
+- Review test artifacts in `/tmp/skill-test-*/`
+- Consult the helper library: `lib/docker-helpers.sh`