Initial commit
This commit is contained in:
Zhongwei Li
392
skills/skill-isolation-tester/test-templates/README.md
Normal file
392
skills/skill-isolation-tester/test-templates/README.md
Normal file
@@ -0,0 +1,392 @@
|
||||
# Skill Test Templates
|
||||
|
||||
Production-ready test templates for validating Claude Code skills in isolated environments.
|
||||
|
||||
## Overview
|
||||
|
||||
These templates provide standardized testing workflows for different skill types. Each template includes:
|
||||
- Pre-flight environment validation
|
||||
- Before/after snapshots for comparison
|
||||
- Comprehensive safety and security checks
|
||||
- Detailed reporting with pass/fail criteria
|
||||
- Automatic cleanup on exit (success or failure)
|
||||
|
||||
## CI/CD Integration with JSON Output
|
||||
|
||||
All test templates support JSON output for integration with CI/CD pipelines. The JSON reporter generates:
|
||||
- **Structured JSON** - Machine-readable test results
|
||||
- **JUnit XML** - Compatible with Jenkins, GitLab CI, GitHub Actions
|
||||
- **Markdown Summary** - Human-readable reports for GitHub Actions
|
||||
|
||||
**Enable JSON output:**
|
||||
```bash
|
||||
export JSON_ENABLED=true
|
||||
./test-templates/docker-skill-test-json.sh my-skill
|
||||
```
|
||||
|
||||
**Output files:**
|
||||
- `test-report.json` - Full structured test data
|
||||
- `test-report.junit.xml` - JUnit format for CI systems
|
||||
- `test-report.md` - Markdown summary
|
||||
|
||||
**JSON Report Structure:**
|
||||
```json
|
||||
{
|
||||
"test_name": "docker-skill-test",
|
||||
"skill_name": "my-skill",
|
||||
"timestamp": "2025-11-02T12:00:00Z",
|
||||
"status": "passed",
|
||||
"duration_seconds": 45,
|
||||
"exit_code": 0,
|
||||
"metrics": {
|
||||
"containers_created": 2,
|
||||
"images_created": 1,
|
||||
"execution_duration_seconds": 12
|
||||
},
|
||||
"issues": [],
|
||||
"recommendations": []
|
||||
}
|
||||
```
|
||||
|
||||
**GitHub Actions Integration:**
|
||||
```yaml
|
||||
- name: Test Skill
|
||||
run: |
|
||||
export JSON_ENABLED=true
|
||||
./test-templates/docker-skill-test-json.sh my-skill
|
||||
|
||||
- name: Upload Test Results
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: test-results
|
||||
path: /tmp/skill-test-*/test-report.*
|
||||
```
|
||||
|
||||
See `lib/json-reporter.sh` for full API documentation.
|
||||
|
||||
---
|
||||
|
||||
## Available Templates
|
||||
|
||||
### 1. Docker Skill Test (`docker-skill-test.sh`)
|
||||
|
||||
**Use for skills that:**
|
||||
- Start or manage Docker containers
|
||||
- Build Docker images
|
||||
- Work with Docker volumes, networks, or compose files
|
||||
- Require Docker daemon access
|
||||
|
||||
**Features:**
|
||||
- Tracks Docker resource creation (containers, images, volumes, networks)
|
||||
- Detects orphaned containers
|
||||
- Validates cleanup behavior
|
||||
- Resource limit enforcement
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
chmod +x test-templates/docker-skill-test.sh
|
||||
./test-templates/docker-skill-test.sh my-docker-skill
|
||||
```
|
||||
|
||||
**Customization:**
|
||||
Edit the skill execution command on line ~178:
|
||||
```bash
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
./skill.sh test-mode # <-- Customize this
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. API Skill Test (`api-skill-test.sh`)
|
||||
|
||||
**Use for skills that:**
|
||||
- Make HTTP/HTTPS requests to external APIs
|
||||
- Require API keys or authentication
|
||||
- Interact with web services
|
||||
- Need network access
|
||||
|
||||
**Features:**
|
||||
- Network traffic monitoring
|
||||
- API call detection and counting
|
||||
- API key/secret leak detection
|
||||
- Rate limiting validation
|
||||
- HTTPS enforcement checking
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
chmod +x test-templates/api-skill-test.sh
|
||||
./test-templates/api-skill-test.sh my-api-skill
|
||||
```
|
||||
|
||||
**Optional: Enable network capture:**
|
||||
```bash
|
||||
# Requires tcpdump and sudo
|
||||
sudo apt-get install tcpdump # or brew install tcpdump
|
||||
./test-templates/api-skill-test.sh my-api-skill
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. File Manipulation Skill Test (`file-manipulation-skill-test.sh`)
|
||||
|
||||
**Use for skills that:**
|
||||
- Create, read, update, or delete files
|
||||
- Modify configuration files
|
||||
- Generate reports or artifacts
|
||||
- Perform filesystem operations
|
||||
|
||||
**Features:**
|
||||
- Complete filesystem diff (added/removed/modified files)
|
||||
- File permission validation
|
||||
- Sensitive data scanning
|
||||
- Temp file cleanup verification
|
||||
- MD5 checksum comparison
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
chmod +x test-templates/file-manipulation-skill-test.sh
|
||||
./test-templates/file-manipulation-skill-test.sh my-file-skill
|
||||
```
|
||||
|
||||
**Customization:**
|
||||
Add your own test files to the workspace (lines 54-70):
|
||||
```bash
|
||||
cat > "$TEST_DIR/test-workspace/your-file.txt" <<'EOF'
|
||||
Your test content here
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Git Skill Test (`git-skill-test.sh`)
|
||||
|
||||
**Use for skills that:**
|
||||
- Create commits, branches, or tags
|
||||
- Modify git history or configuration
|
||||
- Work with git worktrees
|
||||
- Interact with remote repositories
|
||||
|
||||
**Features:**
|
||||
- Git state comparison (commits, branches, tags)
|
||||
- Working tree cleanliness validation
|
||||
- Force operation detection
|
||||
- History rewriting detection
|
||||
- Dangling commit detection
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
chmod +x test-templates/git-skill-test.sh
|
||||
./test-templates/git-skill-test.sh my-git-skill
|
||||
```
|
||||
|
||||
**Customization:**
|
||||
Modify the test repository setup (lines 59-81) to match your skill's requirements.
|
||||
|
||||
---
|
||||
|
||||
## Common Usage Patterns
|
||||
|
||||
### Basic Test Execution
|
||||
|
||||
```bash
|
||||
# Run test for a specific skill
|
||||
./test-templates/docker-skill-test.sh my-skill-name
|
||||
|
||||
# Keep container for debugging
|
||||
export SKILL_TEST_KEEP_CONTAINER="true"
|
||||
./test-templates/docker-skill-test.sh my-skill-name
|
||||
|
||||
# Keep images after test
|
||||
export SKILL_TEST_REMOVE_IMAGES="false"
|
||||
./test-templates/docker-skill-test.sh my-skill-name
|
||||
```
|
||||
|
||||
### Custom Resource Limits
|
||||
|
||||
```bash
|
||||
# Set custom memory/CPU limits
|
||||
export SKILL_TEST_MEMORY_LIMIT="1g"
|
||||
export SKILL_TEST_CPU_LIMIT="2.0"
|
||||
./test-templates/docker-skill-test.sh my-skill-name
|
||||
```
|
||||
|
||||
### Parallel Testing
|
||||
|
||||
```bash
|
||||
# Test multiple skills in parallel
|
||||
for skill in skill1 skill2 skill3; do
|
||||
./test-templates/docker-skill-test.sh "$skill" &
|
||||
done
|
||||
wait
|
||||
echo "All tests complete!"
|
||||
```
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
```bash
|
||||
# Exit code 0 = pass, 1 = fail
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
SKILLS=(
|
||||
"skill-creator"
|
||||
"claude-code-otel-setup"
|
||||
"playwright-e2e-automation"
|
||||
)
|
||||
|
||||
for skill in "${SKILLS[@]}"; do
|
||||
echo "Testing $skill..."
|
||||
./test-templates/docker-skill-test.sh "$skill" || {
|
||||
echo "❌ $skill failed!"
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
echo "✅ All skills passed!"
|
||||
```
|
||||
|
||||
## Customizing Templates
|
||||
|
||||
### Add Custom Validation
|
||||
|
||||
Insert your own checks before the "Generate Test Report" section:
|
||||
|
||||
```bash
|
||||
# ============================================================================
|
||||
# Custom Validation
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Running Custom Checks ==="
|
||||
|
||||
# Your custom checks here
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
# Example: Check if specific file exists
|
||||
test -f /workspace/expected-output.txt || {
|
||||
echo 'ERROR: Expected output file not found'
|
||||
exit 1
|
||||
}
|
||||
"
|
||||
```
|
||||
|
||||
### Modify Execution Command
|
||||
|
||||
Each template has a skill execution section. Customize the command to match your skill's interface:
|
||||
|
||||
```bash
|
||||
# Example: Run skill with arguments
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
./skill.sh --mode=test --output=/workspace/results
|
||||
"
|
||||
|
||||
# Example: Source skill as library
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
source /root/.claude/skills/$SKILL_NAME/lib.sh
|
||||
run_skill_tests
|
||||
"
|
||||
```
|
||||
|
||||
### Add Pre-Test Setup
|
||||
|
||||
Insert setup steps after the "Build Test Environment" section:
|
||||
|
||||
```bash
|
||||
# Install additional dependencies
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
apt-get update && apt-get install -y your-package
|
||||
"
|
||||
|
||||
# Set environment variables
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
export SKILL_CONFIG_PATH=/etc/skill-config.json
|
||||
"
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
All templates support these environment variables:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `SKILL_TEST_KEEP_CONTAINER` | `false` | Keep container after test for debugging |
|
||||
| `SKILL_TEST_REMOVE_IMAGES` | `true` | Remove test images after completion |
|
||||
| `SKILL_TEST_MEMORY_LIMIT` | `512m` | Container memory limit |
|
||||
| `SKILL_TEST_CPU_LIMIT` | `1.0` | Container CPU limit (cores) |
|
||||
| `SKILL_TEST_TEMP_DIR` | `/tmp/skill-test-*` | Temporary directory for test artifacts |
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- `0` - Test passed (skill executed successfully)
|
||||
- `1` - Test failed (skill execution error or validation failure)
|
||||
- `>1` - Other errors (environment setup, Docker issues, etc.)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Docker daemon not running"
|
||||
```bash
|
||||
# macOS
|
||||
open -a Docker
|
||||
|
||||
# Linux
|
||||
sudo systemctl start docker
|
||||
```
|
||||
|
||||
### "Permission denied" errors
|
||||
```bash
|
||||
# Add user to docker group
|
||||
sudo usermod -aG docker $USER
|
||||
newgrp docker
|
||||
```
|
||||
|
||||
### Container hangs or never exits
|
||||
```bash
|
||||
# Set a timeout in your skill execution
|
||||
timeout 300 ./test-templates/docker-skill-test.sh my-skill
|
||||
```
|
||||
|
||||
### Need to inspect failed test
|
||||
```bash
|
||||
# Keep container after failure
|
||||
export SKILL_TEST_KEEP_CONTAINER="true"
|
||||
./test-templates/docker-skill-test.sh my-skill
|
||||
|
||||
# Inspect container
|
||||
docker start -ai <container-id>
|
||||
docker logs <container-id>
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Run tests before committing** - Catch environment-specific bugs early
|
||||
2. **Test in clean environment** - Don't rely on local configs or files
|
||||
3. **Validate cleanup** - Ensure skills don't leave orphaned resources
|
||||
4. **Check for secrets** - Never commit API keys or sensitive data
|
||||
5. **Document dependencies** - List all required packages and tools
|
||||
6. **Use resource limits** - Prevent runaway processes
|
||||
7. **Review diffs carefully** - Understand all file system changes
|
||||
|
||||
## Contributing
|
||||
|
||||
To add a new test template:
|
||||
|
||||
1. Copy an existing template as a starting point
|
||||
2. Customize for your skill type
|
||||
3. Add comprehensive validation checks
|
||||
4. Update this README with usage documentation
|
||||
5. Test your template with at least 3 different skills
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `../lib/docker-helpers.sh` - Shared helper functions
|
||||
- `../modes/mode2-docker.md` - Docker isolation mode documentation
|
||||
- `../skill.md` - Main skill documentation
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- Check the skill logs: `docker logs <container-id>`
|
||||
- Review test artifacts in `/tmp/skill-test-*/`
|
||||
- Consult the helper library: `lib/docker-helpers.sh`
|
||||
317
skills/skill-isolation-tester/test-templates/api-skill-test.sh
Normal file
317
skills/skill-isolation-tester/test-templates/api-skill-test.sh
Normal file
@@ -0,0 +1,317 @@
|
||||
#!/bin/bash
|
||||
# Test Template for API-Calling Skills
|
||||
# Use this template when testing skills that:
|
||||
# - Make HTTP/HTTPS requests to external APIs
|
||||
# - Require API keys or authentication
|
||||
# - Need network access
|
||||
# - Interact with web services
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================================
|
||||
# Configuration
|
||||
# ============================================================================
|
||||
|
||||
SKILL_NAME="${1:-example-api-skill}"
|
||||
SKILL_PATH="$HOME/.claude/skills/$SKILL_NAME"
|
||||
TEST_ID="$(date +%s)"
|
||||
TEST_DIR="/tmp/skill-test-$TEST_ID"
|
||||
|
||||
# ============================================================================
|
||||
# Load Helper Library
|
||||
# ============================================================================
|
||||
|
||||
HELPER_LIB="$HOME/.claude/skills/skill-isolation-tester/lib/docker-helpers.sh"
|
||||
if [[ ! -f "$HELPER_LIB" ]]; then
|
||||
echo "ERROR: Helper library not found: $HELPER_LIB"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "$HELPER_LIB"
|
||||
|
||||
# ============================================================================
|
||||
# Setup Cleanup Trap
|
||||
# ============================================================================
|
||||
|
||||
export SKILL_TEST_TEMP_DIR="$TEST_DIR"
|
||||
export SKILL_TEST_KEEP_CONTAINER="false"
|
||||
export SKILL_TEST_REMOVE_IMAGES="true"
|
||||
|
||||
trap cleanup_on_exit EXIT
|
||||
|
||||
# ============================================================================
|
||||
# Pre-flight Checks
|
||||
# ============================================================================
|
||||
|
||||
echo "=== API Skill Test: $SKILL_NAME ==="
|
||||
echo "Test ID: $TEST_ID"
|
||||
echo ""
|
||||
|
||||
# Validate skill exists
|
||||
if [[ ! -d "$SKILL_PATH" ]]; then
|
||||
echo "ERROR: Skill not found: $SKILL_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate Docker environment
|
||||
preflight_check_docker || exit 1
|
||||
|
||||
# Check internet connectivity
|
||||
if ! curl -s --max-time 5 https://www.google.com > /dev/null 2>&1; then
|
||||
echo "⚠ WARNING: No internet connectivity detected"
|
||||
echo " API skill may fail if it requires external network access"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Build Test Environment
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Building Test Environment ==="
|
||||
|
||||
mkdir -p "$TEST_DIR"
|
||||
|
||||
# Create test Dockerfile
|
||||
cat > "$TEST_DIR/Dockerfile" <<EOF
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Install dependencies for API testing
|
||||
RUN apt-get update && apt-get install -y \\
|
||||
curl \\
|
||||
jq \\
|
||||
ca-certificates \\
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy skill under test
|
||||
COPY skill/ /root/.claude/skills/$SKILL_NAME/
|
||||
|
||||
WORKDIR /root
|
||||
|
||||
CMD ["/bin/bash"]
|
||||
EOF
|
||||
|
||||
# Copy skill to test directory
|
||||
cp -r "$SKILL_PATH" "$TEST_DIR/skill/"
|
||||
|
||||
# Build test image
|
||||
safe_docker_build "$TEST_DIR/Dockerfile" "skill-test:$SKILL_NAME" || {
|
||||
echo "ERROR: Failed to build test image"
|
||||
exit 1
|
||||
}
|
||||
|
||||
export SKILL_TEST_IMAGE_NAME="skill-test:$SKILL_NAME"
|
||||
|
||||
# ============================================================================
|
||||
# Network Monitoring Setup
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Setting Up Network Monitoring ==="
|
||||
|
||||
# Create network monitor log
|
||||
NETWORK_LOG="$TEST_DIR/network-activity.log"
|
||||
touch "$NETWORK_LOG"
|
||||
|
||||
# Start tcpdump in background (if available)
|
||||
if command -v tcpdump &> /dev/null; then
|
||||
echo "Starting network capture..."
|
||||
sudo tcpdump -i any -w "$TEST_DIR/network-capture.pcap" &
|
||||
TCPDUMP_PID=$!
|
||||
echo "tcpdump PID: $TCPDUMP_PID"
|
||||
else
|
||||
echo "tcpdump not available - skipping network capture"
|
||||
TCPDUMP_PID=""
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Run Skill in Container
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Running Skill in Isolated Container ==="
|
||||
|
||||
# Start container with DNS configuration
|
||||
safe_docker_run "skill-test:$SKILL_NAME" \
|
||||
--dns 8.8.8.8 \
|
||||
--dns 8.8.4.4 \
|
||||
bash -c "sleep infinity" || {
|
||||
echo "ERROR: Failed to start test container"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Execute skill and capture network activity
|
||||
echo "Executing skill..."
|
||||
START_TIME=$(date +%s)
|
||||
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
# Add your skill execution command here
|
||||
# Example: ./api-skill.sh --test-mode
|
||||
echo 'Skill execution placeholder - customize this for your skill'
|
||||
|
||||
# Log any curl/wget/http calls made
|
||||
if command -v curl &> /dev/null; then
|
||||
echo 'curl is available in container'
|
||||
fi
|
||||
if command -v wget &> /dev/null; then
|
||||
echo 'wget is available in container'
|
||||
fi
|
||||
" 2>&1 | tee "$NETWORK_LOG" || {
|
||||
EXEC_EXIT_CODE=$?
|
||||
echo "ERROR: Skill execution failed with exit code: $EXEC_EXIT_CODE"
|
||||
|
||||
# Stop network capture
|
||||
if [[ -n "$TCPDUMP_PID" ]]; then
|
||||
sudo kill "$TCPDUMP_PID" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
exit "$EXEC_EXIT_CODE"
|
||||
}
|
||||
|
||||
END_TIME=$(date +%s)
|
||||
EXECUTION_TIME=$((END_TIME - START_TIME))
|
||||
|
||||
# Stop network capture
|
||||
if [[ -n "$TCPDUMP_PID" ]]; then
|
||||
sudo kill "$TCPDUMP_PID" 2>/dev/null || true
|
||||
echo "Network capture saved to: $TEST_DIR/network-capture.pcap"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Analyze Network Activity
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Analyzing Network Activity ==="
|
||||
|
||||
# Check for API calls in logs
|
||||
echo "Searching for HTTP/HTTPS requests..."
|
||||
|
||||
API_CALLS=$(grep -iE "http://|https://|curl|wget|GET|POST|PUT|DELETE" "$NETWORK_LOG" || true)
|
||||
|
||||
if [[ -n "$API_CALLS" ]]; then
|
||||
echo "Detected API calls:"
|
||||
echo "$API_CALLS"
|
||||
|
||||
# Extract unique domains
|
||||
DOMAINS=$(echo "$API_CALLS" | grep -oE "https?://[^/\"]+" | sort -u || true)
|
||||
if [[ -n "$DOMAINS" ]]; then
|
||||
echo ""
|
||||
echo "Unique API endpoints:"
|
||||
echo "$DOMAINS"
|
||||
fi
|
||||
else
|
||||
echo "No obvious API calls detected in logs"
|
||||
fi
|
||||
|
||||
# Check container network stats
|
||||
echo ""
|
||||
echo "Container network statistics:"
|
||||
docker stats --no-stream --format "table {{.Name}}\t{{.NetIO}}" "$SKILL_TEST_CONTAINER_ID"
|
||||
|
||||
# ============================================================================
|
||||
# Validate API Key Handling
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Validating API Key Security ==="
|
||||
|
||||
# Check if API keys appear in logs (security concern)
|
||||
POTENTIAL_KEYS=$(grep -iE "api[-_]?key|token|secret|password|bearer" "$NETWORK_LOG" | grep -v "API_KEY=" || true)
|
||||
|
||||
if [[ -n "$POTENTIAL_KEYS" ]]; then
|
||||
echo "⚠ WARNING: Potential API keys/secrets found in logs:"
|
||||
echo "$POTENTIAL_KEYS"
|
||||
echo ""
|
||||
echo "SECURITY ISSUE: API keys should NOT appear in logs!"
|
||||
echo " - Use environment variables instead"
|
||||
echo " - Redact sensitive data in log output"
|
||||
fi
|
||||
|
||||
# Check for hardcoded endpoints
|
||||
HARDCODED_URLS=$(grep -rn "http://" "$SKILL_PATH" 2>/dev/null | grep -v "example.com" || true)
|
||||
if [[ -n "$HARDCODED_URLS" ]]; then
|
||||
echo "⚠ WARNING: Hardcoded HTTP URLs found (should use HTTPS):"
|
||||
echo "$HARDCODED_URLS"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Rate Limiting Check
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Checking Rate Limiting Behavior ==="
|
||||
|
||||
# Count number of requests made
|
||||
REQUEST_COUNT=$(grep -icE "GET|POST|PUT|DELETE" "$NETWORK_LOG" || echo "0")
|
||||
echo "Total HTTP requests detected: $REQUEST_COUNT"
|
||||
|
||||
if [[ $REQUEST_COUNT -gt 100 ]]; then
|
||||
echo "⚠ WARNING: High number of API requests ($REQUEST_COUNT)"
|
||||
echo " - Consider implementing rate limiting"
|
||||
echo " - Use caching to reduce API calls"
|
||||
echo " - Check for request loops"
|
||||
fi
|
||||
|
||||
REQUESTS_PER_SECOND=$((REQUEST_COUNT / EXECUTION_TIME))
|
||||
echo "Requests per second: $REQUESTS_PER_SECOND"
|
||||
|
||||
# ============================================================================
|
||||
# Generate Test Report
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Test Report ==="
|
||||
echo ""
|
||||
|
||||
CONTAINER_EXIT_CODE=$(get_container_exit_code "$SKILL_TEST_CONTAINER_ID")
|
||||
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
echo "✅ TEST PASSED"
|
||||
else
|
||||
echo "❌ TEST FAILED"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo " - Exit code: $CONTAINER_EXIT_CODE"
|
||||
echo " - Execution time: ${EXECUTION_TIME}s"
|
||||
echo " - API requests: $REQUEST_COUNT"
|
||||
echo " - Network log: $NETWORK_LOG"
|
||||
|
||||
echo ""
|
||||
echo "Security Checklist:"
|
||||
if [[ -z "$POTENTIAL_KEYS" ]]; then
|
||||
echo " ✓ No API keys in logs"
|
||||
else
|
||||
echo " ✗ API keys found in logs"
|
||||
fi
|
||||
|
||||
if [[ -z "$HARDCODED_URLS" ]]; then
|
||||
echo " ✓ No hardcoded HTTP URLs"
|
||||
else
|
||||
echo " ✗ Hardcoded HTTP URLs found"
|
||||
fi
|
||||
|
||||
if [[ $REQUEST_COUNT -lt 100 ]]; then
|
||||
echo " ✓ Reasonable request volume"
|
||||
else
|
||||
echo " ✗ High request volume"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Recommendations:"
|
||||
echo " - Document all external API dependencies"
|
||||
echo " - Implement request caching where possible"
|
||||
echo " - Use exponential backoff for retries"
|
||||
echo " - Respect API rate limits"
|
||||
echo " - Use HTTPS for all API calls"
|
||||
echo " - Never log API keys or secrets"
|
||||
|
||||
# Exit with appropriate code
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,302 @@
|
||||
#!/bin/bash
|
||||
# Test Template for Docker-Based Skills with JSON Output
|
||||
# This is an enhanced version of docker-skill-test.sh with CI/CD integration
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================================
|
||||
# Configuration
|
||||
# ============================================================================
|
||||
|
||||
SKILL_NAME="${1:-example-docker-skill}"
|
||||
SKILL_PATH="$HOME/.claude/skills/$SKILL_NAME"
|
||||
TEST_ID="$(date +%s)"
|
||||
TEST_DIR="/tmp/skill-test-$TEST_ID"
|
||||
|
||||
# JSON reporting
|
||||
export JSON_REPORT_FILE="$TEST_DIR/test-report.json"
|
||||
export JSON_ENABLED="${JSON_ENABLED:-true}"
|
||||
|
||||
# ============================================================================
|
||||
# Load Helper Libraries
|
||||
# ============================================================================
|
||||
|
||||
HELPER_LIB="$HOME/.claude/skills/skill-isolation-tester/lib/docker-helpers.sh"
|
||||
JSON_LIB="$HOME/.claude/skills/skill-isolation-tester/lib/json-reporter.sh"
|
||||
|
||||
if [[ ! -f "$HELPER_LIB" ]]; then
|
||||
echo "ERROR: Helper library not found: $HELPER_LIB"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$JSON_LIB" ]]; then
|
||||
echo "ERROR: JSON reporter library not found: $JSON_LIB"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "$HELPER_LIB"
|
||||
# shellcheck source=/dev/null
|
||||
source "$JSON_LIB"
|
||||
|
||||
# ============================================================================
|
||||
# Setup Cleanup Trap
|
||||
# ============================================================================
|
||||
|
||||
export SKILL_TEST_TEMP_DIR="$TEST_DIR"
|
||||
export SKILL_TEST_KEEP_CONTAINER="false"
|
||||
export SKILL_TEST_REMOVE_IMAGES="true"
|
||||
|
||||
cleanup_and_finalize() {
|
||||
local exit_code=$?
|
||||
local end_time=$(date +%s)
|
||||
local duration=$((end_time - START_TIME))
|
||||
|
||||
# Finalize JSON report
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_finalize "$exit_code" "$duration"
|
||||
export_all_formats "$TEST_DIR/test-report"
|
||||
fi
|
||||
|
||||
# Standard cleanup
|
||||
cleanup_on_exit
|
||||
|
||||
exit "$exit_code"
|
||||
}
|
||||
|
||||
trap cleanup_and_finalize EXIT
|
||||
|
||||
# ============================================================================
|
||||
# Pre-flight Checks
|
||||
# ============================================================================
|
||||
|
||||
echo "=== Docker Skill Test (JSON Mode): $SKILL_NAME ==="
|
||||
echo "Test ID: $TEST_ID"
|
||||
echo ""
|
||||
|
||||
# Create test directory
|
||||
mkdir -p "$TEST_DIR"
|
||||
|
||||
# Initialize JSON report
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_init "docker-skill-test" "$SKILL_NAME"
|
||||
fi
|
||||
|
||||
# Validate skill exists
|
||||
if [[ ! -d "$SKILL_PATH" ]]; then
|
||||
echo "ERROR: Skill not found: $SKILL_PATH"
|
||||
[[ "$JSON_ENABLED" == "true" ]] && json_add_issue "error" "setup" "Skill directory not found: $SKILL_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate Docker environment
|
||||
if ! preflight_check_docker; then
|
||||
[[ "$JSON_ENABLED" == "true" ]] && json_add_issue "error" "environment" "Docker pre-flight checks failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Baseline Measurements (Before)
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Taking Baseline Measurements ==="
|
||||
|
||||
START_TIME=$(date +%s)
|
||||
|
||||
BEFORE_CONTAINERS=$(docker ps -a --format '{{.ID}}' | wc -l)
|
||||
BEFORE_IMAGES=$(docker images --format '{{.ID}}' | wc -l)
|
||||
BEFORE_VOLUMES=$(docker volume ls --format '{{.Name}}' | wc -l)
|
||||
BEFORE_NETWORKS=$(docker network ls --format '{{.ID}}' | wc -l)
|
||||
|
||||
echo "Before test:"
|
||||
echo " Containers: $BEFORE_CONTAINERS"
|
||||
echo " Images: $BEFORE_IMAGES"
|
||||
echo " Volumes: $BEFORE_VOLUMES"
|
||||
echo " Networks: $BEFORE_NETWORKS"
|
||||
|
||||
# Record baseline in JSON
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_add_metric "baseline_containers" "$BEFORE_CONTAINERS"
|
||||
json_add_metric "baseline_images" "$BEFORE_IMAGES"
|
||||
json_add_metric "baseline_volumes" "$BEFORE_VOLUMES"
|
||||
json_add_metric "baseline_networks" "$BEFORE_NETWORKS"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Build Test Environment
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Building Test Environment ==="
|
||||
|
||||
# Create test Dockerfile
|
||||
cat > "$TEST_DIR/Dockerfile" <<EOF
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Install dependencies
|
||||
RUN apt-get update && apt-get install -y \\
|
||||
curl \\
|
||||
git \\
|
||||
nodejs \\
|
||||
npm \\
|
||||
docker.io \\
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Claude Code (mock for testing)
|
||||
RUN mkdir -p /root/.claude/skills
|
||||
|
||||
# Copy skill under test
|
||||
COPY skill/ /root/.claude/skills/$SKILL_NAME/
|
||||
|
||||
WORKDIR /root
|
||||
|
||||
CMD ["/bin/bash"]
|
||||
EOF
|
||||
|
||||
# Copy skill to test directory
|
||||
cp -r "$SKILL_PATH" "$TEST_DIR/skill/"
|
||||
|
||||
# Build test image
|
||||
BUILD_START=$(date +%s)
|
||||
if ! safe_docker_build "$TEST_DIR/Dockerfile" "skill-test:$SKILL_NAME"; then
|
||||
echo "ERROR: Failed to build test image"
|
||||
[[ "$JSON_ENABLED" == "true" ]] && json_add_issue "error" "build" "Docker image build failed"
|
||||
exit 1
|
||||
fi
|
||||
BUILD_END=$(date +%s)
|
||||
BUILD_DURATION=$((BUILD_END - BUILD_START))
|
||||
|
||||
export SKILL_TEST_IMAGE_NAME="skill-test:$SKILL_NAME"
|
||||
|
||||
# Record build metrics
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_add_metric "build_duration_seconds" "$BUILD_DURATION" "seconds"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Run Skill in Container
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Running Skill in Isolated Container ==="
|
||||
|
||||
# Start container with Docker socket access
|
||||
if ! safe_docker_run "skill-test:$SKILL_NAME" \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
bash -c "sleep infinity"; then
|
||||
echo "ERROR: Failed to start test container"
|
||||
[[ "$JSON_ENABLED" == "true" ]] && json_add_issue "error" "runtime" "Container failed to start"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Execute skill
|
||||
echo "Executing skill..."
|
||||
EXEC_START=$(date +%s)
|
||||
|
||||
EXEC_OUTPUT=$(docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
echo 'Skill execution placeholder - customize this for your skill'
|
||||
" 2>&1) || {
|
||||
EXEC_EXIT_CODE=$?
|
||||
echo "ERROR: Skill execution failed with exit code: $EXEC_EXIT_CODE"
|
||||
[[ "$JSON_ENABLED" == "true" ]] && json_add_issue "error" "execution" "Skill failed with exit code $EXEC_EXIT_CODE"
|
||||
exit "$EXEC_EXIT_CODE"
|
||||
}
|
||||
|
||||
EXEC_END=$(date +%s)
|
||||
EXEC_DURATION=$((EXEC_END - EXEC_START))
|
||||
|
||||
# Record execution metrics
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_add_metric "execution_duration_seconds" "$EXEC_DURATION" "seconds"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Collect Measurements (After)
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Collecting Post-Execution Measurements ==="
|
||||
|
||||
sleep 2 # Wait for async operations
|
||||
|
||||
AFTER_CONTAINERS=$(docker ps -a --format '{{.ID}}' | wc -l)
|
||||
AFTER_IMAGES=$(docker images --format '{{.ID}}' | wc -l)
|
||||
AFTER_VOLUMES=$(docker volume ls --format '{{.Name}}' | wc -l)
|
||||
AFTER_NETWORKS=$(docker network ls --format '{{.ID}}' | wc -l)
|
||||
|
||||
CONTAINERS_DELTA=$((AFTER_CONTAINERS - BEFORE_CONTAINERS))
|
||||
IMAGES_DELTA=$((AFTER_IMAGES - BEFORE_IMAGES))
|
||||
VOLUMES_DELTA=$((AFTER_VOLUMES - BEFORE_VOLUMES))
|
||||
NETWORKS_DELTA=$((AFTER_NETWORKS - BEFORE_NETWORKS))
|
||||
|
||||
echo "After test:"
|
||||
echo " Containers: $AFTER_CONTAINERS (delta: $CONTAINERS_DELTA)"
|
||||
echo " Images: $AFTER_IMAGES (delta: $IMAGES_DELTA)"
|
||||
echo " Volumes: $AFTER_VOLUMES (delta: $VOLUMES_DELTA)"
|
||||
echo " Networks: $AFTER_NETWORKS (delta: $NETWORKS_DELTA)"
|
||||
|
||||
# Record changes in JSON
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_add_metric "containers_created" "$CONTAINERS_DELTA"
|
||||
json_add_metric "images_created" "$IMAGES_DELTA"
|
||||
json_add_metric "volumes_created" "$VOLUMES_DELTA"
|
||||
json_add_metric "networks_created" "$NETWORKS_DELTA"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Validate Cleanup Behavior
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Validating Skill Cleanup ==="
|
||||
|
||||
# Check for orphaned containers
|
||||
ORPHANED_CONTAINERS=$(docker ps -a --filter "label=created-by-skill=$SKILL_NAME" --format '{{.ID}}' | wc -l)
|
||||
if [[ $ORPHANED_CONTAINERS -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Skill left $ORPHANED_CONTAINERS orphaned container(s)"
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
json_add_issue "warning" "cleanup" "Found $ORPHANED_CONTAINERS orphaned containers"
|
||||
json_add_recommendation "Cleanup" "Implement automatic container cleanup in skill"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Generate Test Report
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Test Report ==="
|
||||
echo ""
|
||||
|
||||
CONTAINER_EXIT_CODE=$(get_container_exit_code "$SKILL_TEST_CONTAINER_ID")
|
||||
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
echo "✅ TEST PASSED"
|
||||
else
|
||||
echo "❌ TEST FAILED"
|
||||
[[ "$JSON_ENABLED" == "true" ]] && json_add_issue "error" "test-failure" "Container exited with code $CONTAINER_EXIT_CODE"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo " - Exit code: $CONTAINER_EXIT_CODE"
|
||||
echo " - Build duration: ${BUILD_DURATION}s"
|
||||
echo " - Execution duration: ${EXEC_DURATION}s"
|
||||
echo " - Docker resources created: $CONTAINERS_DELTA containers, $IMAGES_DELTA images, $VOLUMES_DELTA volumes, $NETWORKS_DELTA networks"
|
||||
|
||||
if [[ "$JSON_ENABLED" == "true" ]]; then
|
||||
echo ""
|
||||
echo "JSON reports will be generated at:"
|
||||
echo " - $TEST_DIR/test-report.json"
|
||||
echo " - $TEST_DIR/test-report.junit.xml"
|
||||
echo " - $TEST_DIR/test-report.md"
|
||||
fi
|
||||
|
||||
# Exit with appropriate code (cleanup_and_finalize will handle JSON)
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,236 @@
|
||||
#!/bin/bash
|
||||
# Test Template for Docker-Based Skills
|
||||
# Use this template when testing skills that:
|
||||
# - Start Docker containers
|
||||
# - Build Docker images
|
||||
# - Manage Docker volumes/networks
|
||||
# - Require Docker daemon access
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================================
|
||||
# Configuration
|
||||
# ============================================================================
|
||||
|
||||
SKILL_NAME="${1:-example-docker-skill}"
|
||||
SKILL_PATH="$HOME/.claude/skills/$SKILL_NAME"
|
||||
TEST_ID="$(date +%s)"
|
||||
TEST_DIR="/tmp/skill-test-$TEST_ID"
|
||||
|
||||
# ============================================================================
|
||||
# Load Helper Library
|
||||
# ============================================================================
|
||||
|
||||
HELPER_LIB="$HOME/.claude/skills/skill-isolation-tester/lib/docker-helpers.sh"
|
||||
if [[ ! -f "$HELPER_LIB" ]]; then
|
||||
echo "ERROR: Helper library not found: $HELPER_LIB"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "$HELPER_LIB"
|
||||
|
||||
# ============================================================================
|
||||
# Setup Cleanup Trap
|
||||
# ============================================================================
|
||||
|
||||
export SKILL_TEST_TEMP_DIR="$TEST_DIR"
|
||||
export SKILL_TEST_KEEP_CONTAINER="false"
|
||||
export SKILL_TEST_REMOVE_IMAGES="true"
|
||||
|
||||
trap cleanup_on_exit EXIT
|
||||
|
||||
# ============================================================================
|
||||
# Pre-flight Checks
|
||||
# ============================================================================
|
||||
|
||||
echo "=== Docker Skill Test: $SKILL_NAME ==="
|
||||
echo "Test ID: $TEST_ID"
|
||||
echo ""
|
||||
|
||||
# Validate skill exists
|
||||
if [[ ! -d "$SKILL_PATH" ]]; then
|
||||
echo "ERROR: Skill not found: $SKILL_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate Docker environment
|
||||
preflight_check_docker || exit 1
|
||||
|
||||
# ============================================================================
|
||||
# Baseline Measurements (Before)
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Taking Baseline Measurements ==="
|
||||
|
||||
# Count Docker resources before test
|
||||
BEFORE_CONTAINERS=$(docker ps -a --format '{{.ID}}' | wc -l)
|
||||
BEFORE_IMAGES=$(docker images --format '{{.ID}}' | wc -l)
|
||||
BEFORE_VOLUMES=$(docker volume ls --format '{{.Name}}' | wc -l)
|
||||
BEFORE_NETWORKS=$(docker network ls --format '{{.ID}}' | wc -l)
|
||||
|
||||
echo "Before test:"
|
||||
echo " Containers: $BEFORE_CONTAINERS"
|
||||
echo " Images: $BEFORE_IMAGES"
|
||||
echo " Volumes: $BEFORE_VOLUMES"
|
||||
echo " Networks: $BEFORE_NETWORKS"
|
||||
|
||||
# ============================================================================
|
||||
# Build Test Environment
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Building Test Environment ==="
|
||||
|
||||
mkdir -p "$TEST_DIR"
|
||||
|
||||
# Create test Dockerfile
|
||||
cat > "$TEST_DIR/Dockerfile" <<EOF
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Install dependencies
|
||||
RUN apt-get update && apt-get install -y \\
|
||||
curl \\
|
||||
git \\
|
||||
nodejs \\
|
||||
npm \\
|
||||
docker.io \\
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Claude Code (mock for testing)
|
||||
RUN mkdir -p /root/.claude/skills
|
||||
|
||||
# Copy skill under test
|
||||
COPY skill/ /root/.claude/skills/$SKILL_NAME/
|
||||
|
||||
WORKDIR /root
|
||||
|
||||
CMD ["/bin/bash"]
|
||||
EOF
|
||||
|
||||
# Copy skill to test directory
|
||||
cp -r "$SKILL_PATH" "$TEST_DIR/skill/"
|
||||
|
||||
# Build test image
|
||||
safe_docker_build "$TEST_DIR/Dockerfile" "skill-test:$SKILL_NAME" || {
|
||||
echo "ERROR: Failed to build test image"
|
||||
exit 1
|
||||
}
|
||||
|
||||
export SKILL_TEST_IMAGE_NAME="skill-test:$SKILL_NAME"
|
||||
|
||||
# ============================================================================
|
||||
# Run Skill in Container
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Running Skill in Isolated Container ==="
|
||||
|
||||
# Start container with Docker socket access (for Docker-in-Docker skills)
|
||||
safe_docker_run "skill-test:$SKILL_NAME" \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
bash -c "sleep infinity" || {
|
||||
echo "ERROR: Failed to start test container"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Execute skill (customize this command based on your skill's interface)
|
||||
echo "Executing skill..."
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
# Add your skill execution command here
|
||||
# Example: ./skill.sh test-mode
|
||||
echo 'Skill execution placeholder - customize this for your skill'
|
||||
" || {
|
||||
EXEC_EXIT_CODE=$?
|
||||
echo "ERROR: Skill execution failed with exit code: $EXEC_EXIT_CODE"
|
||||
exit "$EXEC_EXIT_CODE"
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# Collect Measurements (After)
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Collecting Post-Execution Measurements ==="
|
||||
|
||||
# Wait for async operations to complete
|
||||
sleep 2
|
||||
|
||||
AFTER_CONTAINERS=$(docker ps -a --format '{{.ID}}' | wc -l)
|
||||
AFTER_IMAGES=$(docker images --format '{{.ID}}' | wc -l)
|
||||
AFTER_VOLUMES=$(docker volume ls --format '{{.Name}}' | wc -l)
|
||||
AFTER_NETWORKS=$(docker network ls --format '{{.ID}}' | wc -l)
|
||||
|
||||
echo "After test:"
|
||||
echo " Containers: $AFTER_CONTAINERS (delta: $((AFTER_CONTAINERS - BEFORE_CONTAINERS)))"
|
||||
echo " Images: $AFTER_IMAGES (delta: $((AFTER_IMAGES - BEFORE_IMAGES)))"
|
||||
echo " Volumes: $AFTER_VOLUMES (delta: $((AFTER_VOLUMES - BEFORE_VOLUMES)))"
|
||||
echo " Networks: $AFTER_NETWORKS (delta: $((AFTER_NETWORKS - BEFORE_NETWORKS)))"
|
||||
|
||||
# ============================================================================
|
||||
# Validate Cleanup Behavior
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Validating Skill Cleanup ==="
|
||||
|
||||
# Check for orphaned containers
|
||||
ORPHANED_CONTAINERS=$(docker ps -a --filter "label=created-by-skill=$SKILL_NAME" --format '{{.ID}}' | wc -l)
|
||||
if [[ $ORPHANED_CONTAINERS -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Skill left $ORPHANED_CONTAINERS orphaned container(s)"
|
||||
docker ps -a --filter "label=created-by-skill=$SKILL_NAME"
|
||||
fi
|
||||
|
||||
# Check for unlabeled containers (potential orphans)
|
||||
SKILL_CONTAINERS=$(docker ps -a --filter "name=$SKILL_NAME" --format '{{.ID}}' | wc -l)
|
||||
if [[ $SKILL_CONTAINERS -gt 1 ]]; then
|
||||
echo "⚠ WARNING: Found $SKILL_CONTAINERS containers with skill name pattern"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Generate Test Report
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Test Report ==="
|
||||
echo ""
|
||||
|
||||
CONTAINER_EXIT_CODE=$(get_container_exit_code "$SKILL_TEST_CONTAINER_ID")
|
||||
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
echo "✅ TEST PASSED"
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo " - Skill executed successfully"
|
||||
echo " - Exit code: 0"
|
||||
echo " - Container cleanup: Will be handled by trap"
|
||||
else
|
||||
echo "❌ TEST FAILED"
|
||||
echo ""
|
||||
echo "Summary:"
|
||||
echo " - Skill execution failed"
|
||||
echo " - Exit code: $CONTAINER_EXIT_CODE"
|
||||
echo " - Check logs: docker logs $SKILL_TEST_CONTAINER_ID"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Docker Resources Created:"
|
||||
echo " - Containers: $((AFTER_CONTAINERS - BEFORE_CONTAINERS))"
|
||||
echo " - Images: $((AFTER_IMAGES - BEFORE_IMAGES))"
|
||||
echo " - Volumes: $((AFTER_VOLUMES - BEFORE_VOLUMES))"
|
||||
echo " - Networks: $((AFTER_NETWORKS - BEFORE_NETWORKS))"
|
||||
|
||||
echo ""
|
||||
echo "Cleanup Instructions:"
|
||||
echo " - Test container will be removed automatically"
|
||||
echo " - To manually clean up: docker rm -f $SKILL_TEST_CONTAINER_ID"
|
||||
echo " - To remove test image: docker rmi skill-test:$SKILL_NAME"
|
||||
|
||||
# Exit with appropriate code
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,360 @@
|
||||
#!/bin/bash
|
||||
# Test Template for File-Manipulation Skills
|
||||
# Use this template when testing skills that:
|
||||
# - Create, read, update, or delete files
|
||||
# - Modify configurations or codebases
|
||||
# - Generate reports or artifacts
|
||||
# - Work with filesystem operations
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================================
|
||||
# Configuration
|
||||
# ============================================================================
|
||||
|
||||
SKILL_NAME="${1:-example-file-skill}"
|
||||
SKILL_PATH="$HOME/.claude/skills/$SKILL_NAME"
|
||||
TEST_ID="$(date +%s)"
|
||||
TEST_DIR="/tmp/skill-test-$TEST_ID"
|
||||
|
||||
# ============================================================================
|
||||
# Load Helper Library
|
||||
# ============================================================================
|
||||
|
||||
HELPER_LIB="$HOME/.claude/skills/skill-isolation-tester/lib/docker-helpers.sh"
|
||||
if [[ ! -f "$HELPER_LIB" ]]; then
|
||||
echo "ERROR: Helper library not found: $HELPER_LIB"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "$HELPER_LIB"
|
||||
|
||||
# ============================================================================
|
||||
# Setup Cleanup Trap
|
||||
# ============================================================================
|
||||
|
||||
export SKILL_TEST_TEMP_DIR="$TEST_DIR"
|
||||
export SKILL_TEST_KEEP_CONTAINER="false"
|
||||
export SKILL_TEST_REMOVE_IMAGES="true"
|
||||
|
||||
trap cleanup_on_exit EXIT
|
||||
|
||||
# ============================================================================
|
||||
# Pre-flight Checks
|
||||
# ============================================================================
|
||||
|
||||
echo "=== File Manipulation Skill Test: $SKILL_NAME ==="
|
||||
echo "Test ID: $TEST_ID"
|
||||
echo ""
|
||||
|
||||
# Validate skill exists
|
||||
if [[ ! -d "$SKILL_PATH" ]]; then
|
||||
echo "ERROR: Skill not found: $SKILL_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate Docker environment
|
||||
preflight_check_docker || exit 1
|
||||
|
||||
# ============================================================================
|
||||
# Build Test Environment with Sample Files
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Building Test Environment ==="
|
||||
|
||||
mkdir -p "$TEST_DIR/test-workspace"
|
||||
|
||||
# Create sample files for the skill to manipulate
|
||||
cat > "$TEST_DIR/test-workspace/sample.txt" <<'EOF'
|
||||
This is a sample text file for testing.
|
||||
Line 2
|
||||
Line 3
|
||||
EOF
|
||||
|
||||
cat > "$TEST_DIR/test-workspace/config.json" <<'EOF'
|
||||
{
|
||||
"setting1": "value1",
|
||||
"setting2": 42,
|
||||
"enabled": true
|
||||
}
|
||||
EOF
|
||||
|
||||
mkdir -p "$TEST_DIR/test-workspace/subdir"
|
||||
echo "Nested file" > "$TEST_DIR/test-workspace/subdir/nested.txt"
|
||||
|
||||
# Create Dockerfile
|
||||
cat > "$TEST_DIR/Dockerfile" <<EOF
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Install file manipulation tools
|
||||
RUN apt-get update && apt-get install -y \\
|
||||
coreutils \\
|
||||
jq \\
|
||||
tree \\
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Create workspace
|
||||
RUN mkdir -p /workspace
|
||||
|
||||
# Copy skill
|
||||
COPY skill/ /root/.claude/skills/$SKILL_NAME/
|
||||
|
||||
# Copy test files
|
||||
COPY test-workspace/ /workspace/
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
CMD ["/bin/bash"]
|
||||
EOF
|
||||
|
||||
# Copy skill to test directory
|
||||
cp -r "$SKILL_PATH" "$TEST_DIR/skill/"
|
||||
|
||||
# Build test image
|
||||
safe_docker_build "$TEST_DIR/Dockerfile" "skill-test:$SKILL_NAME" || {
|
||||
echo "ERROR: Failed to build test image"
|
||||
exit 1
|
||||
}
|
||||
|
||||
export SKILL_TEST_IMAGE_NAME="skill-test:$SKILL_NAME"
|
||||
|
||||
# ============================================================================
|
||||
# Take "Before" Filesystem Snapshot
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Taking Filesystem Snapshot (Before) ==="
|
||||
|
||||
# Start container
|
||||
safe_docker_run "skill-test:$SKILL_NAME" bash -c "sleep infinity" || {
|
||||
echo "ERROR: Failed to start test container"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Get baseline file list
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" find /workspace -type f -o -type d | sort > "$TEST_DIR/before-files.txt"
|
||||
|
||||
# Get file sizes and checksums
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /workspace
|
||||
find . -type f -exec md5sum {} \; | sort
|
||||
" > "$TEST_DIR/before-checksums.txt"
|
||||
|
||||
# Count files
|
||||
BEFORE_FILE_COUNT=$(docker exec "$SKILL_TEST_CONTAINER_ID" find /workspace -type f | wc -l)
|
||||
BEFORE_DIR_COUNT=$(docker exec "$SKILL_TEST_CONTAINER_ID" find /workspace -type d | wc -l)
|
||||
|
||||
echo "Before execution:"
|
||||
echo " Files: $BEFORE_FILE_COUNT"
|
||||
echo " Directories: $BEFORE_DIR_COUNT"
|
||||
|
||||
# ============================================================================
|
||||
# Run Skill in Container
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Running Skill in Isolated Container ==="
|
||||
|
||||
# Execute skill
|
||||
echo "Executing skill..."
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
# Add your skill execution command here
|
||||
# Example: ./file-processor.sh /workspace
|
||||
echo 'Skill execution placeholder - customize this for your skill'
|
||||
" || {
|
||||
EXEC_EXIT_CODE=$?
|
||||
echo "ERROR: Skill execution failed with exit code: $EXEC_EXIT_CODE"
|
||||
exit "$EXEC_EXIT_CODE"
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# Take "After" Filesystem Snapshot
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Taking Filesystem Snapshot (After) ==="
|
||||
|
||||
# Get updated file list
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" find /workspace -type f -o -type d | sort > "$TEST_DIR/after-files.txt"
|
||||
|
||||
# Get updated checksums
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /workspace
|
||||
find . -type f -exec md5sum {} \; | sort
|
||||
" > "$TEST_DIR/after-checksums.txt"
|
||||
|
||||
# Count files
|
||||
AFTER_FILE_COUNT=$(docker exec "$SKILL_TEST_CONTAINER_ID" find /workspace -type f | wc -l)
|
||||
AFTER_DIR_COUNT=$(docker exec "$SKILL_TEST_CONTAINER_ID" find /workspace -type d | wc -l)
|
||||
|
||||
echo "After execution:"
|
||||
echo " Files: $AFTER_FILE_COUNT"
|
||||
echo " Directories: $AFTER_DIR_COUNT"
|
||||
|
||||
# ============================================================================
|
||||
# Analyze Filesystem Changes
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Analyzing Filesystem Changes ==="
|
||||
|
||||
# Files added
|
||||
echo ""
|
||||
echo "Files Added:"
|
||||
comm -13 "$TEST_DIR/before-files.txt" "$TEST_DIR/after-files.txt" > "$TEST_DIR/files-added.txt"
|
||||
ADDED_COUNT=$(wc -l < "$TEST_DIR/files-added.txt")
|
||||
echo " Count: $ADDED_COUNT"
|
||||
if [[ $ADDED_COUNT -gt 0 ]]; then
|
||||
head -10 "$TEST_DIR/files-added.txt"
|
||||
if [[ $ADDED_COUNT -gt 10 ]]; then
|
||||
echo " ... and $((ADDED_COUNT - 10)) more"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Files removed
|
||||
echo ""
|
||||
echo "Files Removed:"
|
||||
comm -23 "$TEST_DIR/before-files.txt" "$TEST_DIR/after-files.txt" > "$TEST_DIR/files-removed.txt"
|
||||
REMOVED_COUNT=$(wc -l < "$TEST_DIR/files-removed.txt")
|
||||
echo " Count: $REMOVED_COUNT"
|
||||
if [[ $REMOVED_COUNT -gt 0 ]]; then
|
||||
head -10 "$TEST_DIR/files-removed.txt"
|
||||
if [[ $REMOVED_COUNT -gt 10 ]]; then
|
||||
echo " ... and $((REMOVED_COUNT - 10)) more"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Files modified
|
||||
echo ""
|
||||
echo "Files Modified:"
|
||||
comm -12 "$TEST_DIR/before-files.txt" "$TEST_DIR/after-files.txt" | while read -r file; do
|
||||
BEFORE_HASH=$(grep "$file" "$TEST_DIR/before-checksums.txt" 2>/dev/null | awk '{print $1}' || echo "")
|
||||
AFTER_HASH=$(grep "$file" "$TEST_DIR/after-checksums.txt" 2>/dev/null | awk '{print $1}' || echo "")
|
||||
|
||||
if [[ -n "$BEFORE_HASH" && -n "$AFTER_HASH" && "$BEFORE_HASH" != "$AFTER_HASH" ]]; then
|
||||
echo " $file"
|
||||
fi
|
||||
done | tee "$TEST_DIR/files-modified.txt"
|
||||
MODIFIED_COUNT=$(wc -l < "$TEST_DIR/files-modified.txt")
|
||||
echo " Count: $MODIFIED_COUNT"
|
||||
|
||||
# ============================================================================
|
||||
# Validate File Permissions
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Checking File Permissions ==="
|
||||
|
||||
# Find files with unusual permissions
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
find /workspace -type f -perm /111 -ls
|
||||
" > "$TEST_DIR/executable-files.txt" || true
|
||||
|
||||
EXECUTABLE_COUNT=$(wc -l < "$TEST_DIR/executable-files.txt")
|
||||
if [[ $EXECUTABLE_COUNT -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Found $EXECUTABLE_COUNT executable files"
|
||||
cat "$TEST_DIR/executable-files.txt"
|
||||
else
|
||||
echo "✓ No unexpected executable files"
|
||||
fi
|
||||
|
||||
# Check for world-writable files
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
find /workspace -type f -perm -002 -ls
|
||||
" > "$TEST_DIR/world-writable-files.txt" || true
|
||||
|
||||
WRITABLE_COUNT=$(wc -l < "$TEST_DIR/world-writable-files.txt")
|
||||
if [[ $WRITABLE_COUNT -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Found $WRITABLE_COUNT world-writable files (security risk)"
|
||||
cat "$TEST_DIR/world-writable-files.txt"
|
||||
else
|
||||
echo "✓ No world-writable files"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Check for Sensitive Data
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Scanning for Sensitive Data ==="
|
||||
|
||||
# Check for potential secrets in new files
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
grep -rni 'password\|api[-_]key\|secret\|token' /workspace
|
||||
" 2>/dev/null | tee "$TEST_DIR/potential-secrets.txt" || true
|
||||
|
||||
SECRET_COUNT=$(wc -l < "$TEST_DIR/potential-secrets.txt")
|
||||
if [[ $SECRET_COUNT -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Found $SECRET_COUNT lines with potential secrets"
|
||||
echo " Review: $TEST_DIR/potential-secrets.txt"
|
||||
else
|
||||
echo "✓ No obvious secrets detected"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Validate Cleanup Behavior
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Validating Cleanup Behavior ==="
|
||||
|
||||
# Check for leftover temp files
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
find /tmp -name '*skill*' -o -name '*.tmp' -o -name '*.temp'
|
||||
" > "$TEST_DIR/temp-files.txt" || true
|
||||
|
||||
TEMP_COUNT=$(wc -l < "$TEST_DIR/temp-files.txt")
|
||||
if [[ $TEMP_COUNT -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Found $TEMP_COUNT leftover temp files"
|
||||
cat "$TEST_DIR/temp-files.txt"
|
||||
else
|
||||
echo "✓ No leftover temp files"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Generate Test Report
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Test Report ==="
|
||||
echo ""
|
||||
|
||||
CONTAINER_EXIT_CODE=$(get_container_exit_code "$SKILL_TEST_CONTAINER_ID")
|
||||
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
echo "✅ TEST PASSED"
|
||||
else
|
||||
echo "❌ TEST FAILED"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Filesystem Changes Summary:"
|
||||
echo " - Files added: $ADDED_COUNT"
|
||||
echo " - Files removed: $REMOVED_COUNT"
|
||||
echo " - Files modified: $MODIFIED_COUNT"
|
||||
echo " - Total file count change: $((AFTER_FILE_COUNT - BEFORE_FILE_COUNT))"
|
||||
|
||||
echo ""
|
||||
echo "Security & Quality Checklist:"
|
||||
[[ $EXECUTABLE_COUNT -eq 0 ]] && echo " ✓ No unexpected executable files" || echo " ✗ Found executable files"
|
||||
[[ $WRITABLE_COUNT -eq 0 ]] && echo " ✓ No world-writable files" || echo " ✗ Found world-writable files"
|
||||
[[ $SECRET_COUNT -eq 0 ]] && echo " ✓ No secrets in files" || echo " ✗ Potential secrets found"
|
||||
[[ $TEMP_COUNT -eq 0 ]] && echo " ✓ Clean temp directory" || echo " ✗ Leftover temp files"
|
||||
|
||||
echo ""
|
||||
echo "Detailed Reports:"
|
||||
echo " - Files added: $TEST_DIR/files-added.txt"
|
||||
echo " - Files removed: $TEST_DIR/files-removed.txt"
|
||||
echo " - Files modified: $TEST_DIR/files-modified.txt"
|
||||
echo " - Before snapshot: $TEST_DIR/before-files.txt"
|
||||
echo " - After snapshot: $TEST_DIR/after-files.txt"
|
||||
|
||||
# Exit with appropriate code
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
395
skills/skill-isolation-tester/test-templates/git-skill-test.sh
Normal file
395
skills/skill-isolation-tester/test-templates/git-skill-test.sh
Normal file
@@ -0,0 +1,395 @@
|
||||
#!/bin/bash
|
||||
# Test Template for Git-Operation Skills
|
||||
# Use this template when testing skills that:
|
||||
# - Create commits, branches, or tags
|
||||
# - Modify git history or configuration
|
||||
# - Work with git worktrees
|
||||
# - Interact with remote repositories
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ============================================================================
|
||||
# Configuration
|
||||
# ============================================================================
|
||||
|
||||
SKILL_NAME="${1:-example-git-skill}"
|
||||
SKILL_PATH="$HOME/.claude/skills/$SKILL_NAME"
|
||||
TEST_ID="$(date +%s)"
|
||||
TEST_DIR="/tmp/skill-test-$TEST_ID"
|
||||
|
||||
# ============================================================================
|
||||
# Load Helper Library
|
||||
# ============================================================================
|
||||
|
||||
HELPER_LIB="$HOME/.claude/skills/skill-isolation-tester/lib/docker-helpers.sh"
|
||||
if [[ ! -f "$HELPER_LIB" ]]; then
|
||||
echo "ERROR: Helper library not found: $HELPER_LIB"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# shellcheck source=/dev/null
|
||||
source "$HELPER_LIB"
|
||||
|
||||
# ============================================================================
|
||||
# Setup Cleanup Trap
|
||||
# ============================================================================
|
||||
|
||||
export SKILL_TEST_TEMP_DIR="$TEST_DIR"
|
||||
export SKILL_TEST_KEEP_CONTAINER="false"
|
||||
export SKILL_TEST_REMOVE_IMAGES="true"
|
||||
|
||||
trap cleanup_on_exit EXIT
|
||||
|
||||
# ============================================================================
|
||||
# Pre-flight Checks
|
||||
# ============================================================================
|
||||
|
||||
echo "=== Git Skill Test: $SKILL_NAME ==="
|
||||
echo "Test ID: $TEST_ID"
|
||||
echo ""
|
||||
|
||||
# Validate skill exists
|
||||
if [[ ! -d "$SKILL_PATH" ]]; then
|
||||
echo "ERROR: Skill not found: $SKILL_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate Docker environment
|
||||
preflight_check_docker || exit 1
|
||||
|
||||
# ============================================================================
|
||||
# Create Test Git Repository
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Creating Test Git Repository ==="
|
||||
|
||||
mkdir -p "$TEST_DIR/test-repo"
|
||||
cd "$TEST_DIR/test-repo"
|
||||
|
||||
# Initialize git repo
|
||||
git init
|
||||
git config user.name "Test User"
|
||||
git config user.email "test@example.com"
|
||||
|
||||
# Create initial commit
|
||||
echo "# Test Repository" > README.md
|
||||
echo "Initial content" > file1.txt
|
||||
git add .
|
||||
git commit -m "Initial commit"
|
||||
|
||||
# Create a branch
|
||||
git checkout -b feature-branch
|
||||
echo "Feature content" > feature.txt
|
||||
git add feature.txt
|
||||
git commit -m "Add feature"
|
||||
|
||||
# Go back to main
|
||||
git checkout main
|
||||
|
||||
# Create a tag
|
||||
git tag v1.0.0
|
||||
|
||||
echo "Test repository created:"
|
||||
git log --oneline --all --graph
|
||||
echo ""
|
||||
git branch -a
|
||||
echo ""
|
||||
git tag
|
||||
|
||||
# ============================================================================
|
||||
# Build Test Environment
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Building Test Environment ==="
|
||||
|
||||
cd "$TEST_DIR"
|
||||
|
||||
# Create Dockerfile
|
||||
cat > "$TEST_DIR/Dockerfile" <<EOF
|
||||
FROM ubuntu:22.04
|
||||
|
||||
# Install git
|
||||
RUN apt-get update && apt-get install -y \\
|
||||
git \\
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Configure git
|
||||
RUN git config --global user.name "Test User" && \\
|
||||
git config --global user.email "test@example.com"
|
||||
|
||||
# Copy skill
|
||||
COPY skill/ /root/.claude/skills/$SKILL_NAME/
|
||||
|
||||
# Copy test repository
|
||||
COPY test-repo/ /workspace/
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
CMD ["/bin/bash"]
|
||||
EOF
|
||||
|
||||
# Copy skill
|
||||
cp -r "$SKILL_PATH" "$TEST_DIR/skill/"
|
||||
|
||||
# Build test image
|
||||
safe_docker_build "$TEST_DIR/Dockerfile" "skill-test:$SKILL_NAME" || {
|
||||
echo "ERROR: Failed to build test image"
|
||||
exit 1
|
||||
}
|
||||
|
||||
export SKILL_TEST_IMAGE_NAME="skill-test:$SKILL_NAME"
|
||||
|
||||
# ============================================================================
|
||||
# Take "Before" Git Snapshot
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Taking Git Snapshot (Before) ==="
|
||||
|
||||
# Start container
|
||||
safe_docker_run "skill-test:$SKILL_NAME" bash -c "sleep infinity" || {
|
||||
echo "ERROR: Failed to start test container"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Capture git state before
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /workspace
|
||||
git log --all --oneline --graph > /tmp/before-log.txt
|
||||
git branch -a > /tmp/before-branches.txt
|
||||
git tag > /tmp/before-tags.txt
|
||||
git status > /tmp/before-status.txt
|
||||
git config --list > /tmp/before-config.txt
|
||||
" || true
|
||||
|
||||
# Copy snapshots out
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/before-log.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/before-branches.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/before-tags.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/before-status.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/before-config.txt" "$TEST_DIR/"
|
||||
|
||||
BEFORE_COMMIT_COUNT=$(docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "cd /workspace && git rev-list --all --count")
|
||||
BEFORE_BRANCH_COUNT=$(wc -l < "$TEST_DIR/before-branches.txt")
|
||||
BEFORE_TAG_COUNT=$(wc -l < "$TEST_DIR/before-tags.txt")
|
||||
|
||||
echo "Before execution:"
|
||||
echo " Commits: $BEFORE_COMMIT_COUNT"
|
||||
echo " Branches: $BEFORE_BRANCH_COUNT"
|
||||
echo " Tags: $BEFORE_TAG_COUNT"
|
||||
|
||||
# ============================================================================
|
||||
# Run Skill in Container
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Running Skill in Isolated Container ==="
|
||||
|
||||
# Execute skill
|
||||
echo "Executing skill..."
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /root/.claude/skills/$SKILL_NAME
|
||||
# Add your skill execution command here
|
||||
# Example: ./git-skill.sh /workspace
|
||||
echo 'Skill execution placeholder - customize this for your skill'
|
||||
" || {
|
||||
EXEC_EXIT_CODE=$?
|
||||
echo "ERROR: Skill execution failed with exit code: $EXEC_EXIT_CODE"
|
||||
exit "$EXEC_EXIT_CODE"
|
||||
}
|
||||
|
||||
# ============================================================================
|
||||
# Take "After" Git Snapshot
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Taking Git Snapshot (After) ==="
|
||||
|
||||
# Capture git state after
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /workspace
|
||||
git log --all --oneline --graph > /tmp/after-log.txt
|
||||
git branch -a > /tmp/after-branches.txt
|
||||
git tag > /tmp/after-tags.txt
|
||||
git status > /tmp/after-status.txt
|
||||
git config --list > /tmp/after-config.txt
|
||||
" || true
|
||||
|
||||
# Copy snapshots out
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/after-log.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/after-branches.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/after-tags.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/after-status.txt" "$TEST_DIR/"
|
||||
docker cp "$SKILL_TEST_CONTAINER_ID:/tmp/after-config.txt" "$TEST_DIR/"
|
||||
|
||||
AFTER_COMMIT_COUNT=$(docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "cd /workspace && git rev-list --all --count")
|
||||
AFTER_BRANCH_COUNT=$(wc -l < "$TEST_DIR/after-branches.txt")
|
||||
AFTER_TAG_COUNT=$(wc -l < "$TEST_DIR/after-tags.txt")
|
||||
|
||||
echo "After execution:"
|
||||
echo " Commits: $AFTER_COMMIT_COUNT"
|
||||
echo " Branches: $AFTER_BRANCH_COUNT"
|
||||
echo " Tags: $AFTER_TAG_COUNT"
|
||||
|
||||
# ============================================================================
|
||||
# Analyze Git Changes
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Analyzing Git Changes ==="
|
||||
|
||||
# New commits
|
||||
COMMIT_DIFF=$((AFTER_COMMIT_COUNT - BEFORE_COMMIT_COUNT))
|
||||
if [[ $COMMIT_DIFF -gt 0 ]]; then
|
||||
echo "✓ Added $COMMIT_DIFF new commit(s)"
|
||||
|
||||
echo ""
|
||||
echo "New commits:"
|
||||
docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "
|
||||
cd /workspace
|
||||
git log --oneline -n $COMMIT_DIFF
|
||||
"
|
||||
else
|
||||
echo "No new commits created"
|
||||
fi
|
||||
|
||||
# New branches
|
||||
echo ""
|
||||
echo "Branch Changes:"
|
||||
comm -13 "$TEST_DIR/before-branches.txt" "$TEST_DIR/after-branches.txt" > "$TEST_DIR/branches-added.txt"
|
||||
BRANCH_ADDED=$(wc -l < "$TEST_DIR/branches-added.txt")
|
||||
if [[ $BRANCH_ADDED -gt 0 ]]; then
|
||||
echo " Added $BRANCH_ADDED branch(es):"
|
||||
cat "$TEST_DIR/branches-added.txt"
|
||||
fi
|
||||
|
||||
comm -23 "$TEST_DIR/before-branches.txt" "$TEST_DIR/after-branches.txt" > "$TEST_DIR/branches-removed.txt"
|
||||
BRANCH_REMOVED=$(wc -l < "$TEST_DIR/branches-removed.txt")
|
||||
if [[ $BRANCH_REMOVED -gt 0 ]]; then
|
||||
echo " Removed $BRANCH_REMOVED branch(es):"
|
||||
cat "$TEST_DIR/branches-removed.txt"
|
||||
fi
|
||||
|
||||
if [[ $BRANCH_ADDED -eq 0 && $BRANCH_REMOVED -eq 0 ]]; then
|
||||
echo " No branch changes"
|
||||
fi
|
||||
|
||||
# New tags
|
||||
echo ""
|
||||
echo "Tag Changes:"
|
||||
comm -13 "$TEST_DIR/before-tags.txt" "$TEST_DIR/after-tags.txt" > "$TEST_DIR/tags-added.txt"
|
||||
TAG_ADDED=$(wc -l < "$TEST_DIR/tags-added.txt")
|
||||
if [[ $TAG_ADDED -gt 0 ]]; then
|
||||
echo " Added $TAG_ADDED tag(s):"
|
||||
cat "$TEST_DIR/tags-added.txt"
|
||||
fi
|
||||
|
||||
# Config changes
|
||||
echo ""
|
||||
echo "Git Config Changes:"
|
||||
diff "$TEST_DIR/before-config.txt" "$TEST_DIR/after-config.txt" > "$TEST_DIR/config-diff.txt" || true
|
||||
if [[ -s "$TEST_DIR/config-diff.txt" ]]; then
|
||||
echo " Configuration was modified:"
|
||||
cat "$TEST_DIR/config-diff.txt"
|
||||
else
|
||||
echo " No configuration changes"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Check Working Tree Status
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Checking Working Tree Status ==="
|
||||
|
||||
UNCOMMITTED_CHANGES=$(docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "cd /workspace && git status --porcelain" || echo "")
|
||||
if [[ -n "$UNCOMMITTED_CHANGES" ]]; then
|
||||
echo "⚠ WARNING: Uncommitted changes detected:"
|
||||
echo "$UNCOMMITTED_CHANGES"
|
||||
echo ""
|
||||
echo "Skills should clean up working tree after execution!"
|
||||
else
|
||||
echo "✓ Working tree is clean"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Validate Git Safety
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Git Safety Checks ==="
|
||||
|
||||
# Check for force operations in logs
|
||||
docker logs "$SKILL_TEST_CONTAINER_ID" 2>&1 | grep -i "force\|--force\|-f" > "$TEST_DIR/force-operations.txt" || true
|
||||
FORCE_OPS=$(wc -l < "$TEST_DIR/force-operations.txt")
|
||||
if [[ $FORCE_OPS -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Detected $FORCE_OPS force operations"
|
||||
cat "$TEST_DIR/force-operations.txt"
|
||||
else
|
||||
echo "✓ No force operations detected"
|
||||
fi
|
||||
|
||||
# Check for history rewriting
|
||||
docker logs "$SKILL_TEST_CONTAINER_ID" 2>&1 | grep -i "rebase\|reset --hard\|filter-branch" > "$TEST_DIR/history-rewrites.txt" || true
|
||||
REWRITES=$(wc -l < "$TEST_DIR/history-rewrites.txt")
|
||||
if [[ $REWRITES -gt 0 ]]; then
|
||||
echo "⚠ WARNING: Detected $REWRITES history rewrite operations"
|
||||
cat "$TEST_DIR/history-rewrites.txt"
|
||||
else
|
||||
echo "✓ No history rewriting detected"
|
||||
fi
|
||||
|
||||
# Check for dangling commits
|
||||
DANGLING_COMMITS=$(docker exec "$SKILL_TEST_CONTAINER_ID" bash -c "cd /workspace && git fsck --lost-found 2>&1 | grep 'dangling commit'" || echo "")
|
||||
if [[ -n "$DANGLING_COMMITS" ]]; then
|
||||
echo "⚠ WARNING: Dangling commits found (potential data loss)"
|
||||
echo "$DANGLING_COMMITS"
|
||||
else
|
||||
echo "✓ No dangling commits"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# Generate Test Report
|
||||
# ============================================================================
|
||||
|
||||
echo ""
|
||||
echo "=== Test Report ==="
|
||||
echo ""
|
||||
|
||||
CONTAINER_EXIT_CODE=$(get_container_exit_code "$SKILL_TEST_CONTAINER_ID")
|
||||
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
echo "✅ TEST PASSED"
|
||||
else
|
||||
echo "❌ TEST FAILED"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "Git Changes Summary:"
|
||||
echo " - Commits added: $COMMIT_DIFF"
|
||||
echo " - Branches added: $BRANCH_ADDED"
|
||||
echo " - Branches removed: $BRANCH_REMOVED"
|
||||
echo " - Tags added: $TAG_ADDED"
|
||||
|
||||
echo ""
|
||||
echo "Safety Checklist:"
|
||||
[[ -z "$UNCOMMITTED_CHANGES" ]] && echo " ✓ Clean working tree" || echo " ✗ Uncommitted changes"
|
||||
[[ $FORCE_OPS -eq 0 ]] && echo " ✓ No force operations" || echo " ✗ Force operations detected"
|
||||
[[ $REWRITES -eq 0 ]] && echo " ✓ No history rewriting" || echo " ✗ History rewriting detected"
|
||||
[[ -z "$DANGLING_COMMITS" ]] && echo " ✓ No dangling commits" || echo " ✗ Dangling commits found"
|
||||
|
||||
echo ""
|
||||
echo "Detailed Snapshots:"
|
||||
echo " - Before log: $TEST_DIR/before-log.txt"
|
||||
echo " - After log: $TEST_DIR/after-log.txt"
|
||||
echo " - Branch changes: $TEST_DIR/branches-added.txt"
|
||||
echo " - Config diff: $TEST_DIR/config-diff.txt"
|
||||
|
||||
# Exit with appropriate code
|
||||
if [[ $CONTAINER_EXIT_CODE -eq 0 ]]; then
|
||||
exit 0
|
||||
else
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user