Initial commit
This commit is contained in:
391
skills/meta/skill-isolation-tester/data/risk-assessment.md
Normal file
391
skills/meta/skill-isolation-tester/data/risk-assessment.md
Normal file
@@ -0,0 +1,391 @@
|
||||
# Skill Risk Assessment Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide helps you assess the risk level of a skill to determine the appropriate isolation environment for testing. Risk assessment prevents over-isolation (wasting time) and under-isolation (security issues).
|
||||
|
||||
## Risk Levels
|
||||
|
||||
### Low Risk → Git Worktree
|
||||
|
||||
**Characteristics:**
|
||||
- Read-only operations on existing files
|
||||
- No system commands (bash, npm, apt, etc.)
|
||||
- No file creation outside skill directory
|
||||
- No network requests
|
||||
- Pure data processing or analysis
|
||||
- File reading and reporting only
|
||||
|
||||
**Examples:**
|
||||
- Code analyzer that reads files and generates reports
|
||||
- Configuration validator that checks syntax
|
||||
- Documentation generator from code comments
|
||||
- Markdown formatter or linter
|
||||
- Log file parser
|
||||
|
||||
**Appropriate Environment:** Git Worktree (fast, lightweight)
|
||||
|
||||
### Medium Risk → Docker
|
||||
|
||||
**Characteristics:**
|
||||
- File creation in user directories
|
||||
- NPM/pip package installation
|
||||
- Bash commands for file operations
|
||||
- Git operations (clone, commit, etc.)
|
||||
- Network requests (API calls, downloads)
|
||||
- Environment variable reads
|
||||
- Temporary file creation
|
||||
- Database connections (local)
|
||||
|
||||
**Examples:**
|
||||
- Code generator that creates new files
|
||||
- Package installer or dependency manager
|
||||
- API integration that fetches remote data
|
||||
- Build tool that compiles code
|
||||
- Test runner that executes tests
|
||||
- Migration tool that updates files
|
||||
|
||||
**Appropriate Environment:** Docker (OS isolation, reproducible)
|
||||
|
||||
### High Risk → VM
|
||||
|
||||
**Characteristics:**
|
||||
- System configuration changes (/etc/ modifications)
|
||||
- Service installation (systemd, cron)
|
||||
- Kernel module loading
|
||||
- VM or container operations
|
||||
- Database schema migrations (production)
|
||||
- Destructive operations (file deletion, disk formatting)
|
||||
- Privilege escalation (sudo commands)
|
||||
- Unknown or untrusted source
|
||||
|
||||
**Examples:**
|
||||
- System setup automation
|
||||
- Infrastructure provisioning
|
||||
- VM management tools
|
||||
- Security testing tools
|
||||
- Experimental or unreviewed skills
|
||||
- Skills from external repositories
|
||||
|
||||
**Appropriate Environment:** VM (complete isolation, safest)
|
||||
|
||||
## Assessment Checklist
|
||||
|
||||
### Step 1: Parse Skill Manifest (SKILL.md)
|
||||
|
||||
Read the skill's SKILL.md and look for these keywords:
|
||||
|
||||
**Low Risk Indicators:**
|
||||
- "analyze", "read", "parse", "validate", "check", "lint", "format"
|
||||
- "generate report", "calculate", "summarize"
|
||||
- Read-only file operations
|
||||
- No system commands mentioned
|
||||
|
||||
**Medium Risk Indicators:**
|
||||
- "install", "create", "write", "modify", "update", "build", "compile"
|
||||
- "npm install", "pip install", "git clone"
|
||||
- "fetch", "download", "API call"
|
||||
- File creation mentioned
|
||||
- Bash commands for file operations
|
||||
|
||||
**High Risk Indicators:**
|
||||
- "sudo", "systemctl", "cron", "service"
|
||||
- "configure system", "modify /etc"
|
||||
- "VM", "docker run", "container"
|
||||
- "delete", "remove", "format"
|
||||
- "root access", "privilege"
|
||||
|
||||
### Step 2: Scan Skill Code
|
||||
|
||||
If skill includes scripts or code files, scan for:
|
||||
|
||||
**Red Flags (High Risk):**
|
||||
```bash
|
||||
# In bash scripts
|
||||
sudo
|
||||
systemctl
|
||||
/etc/
|
||||
chmod 777
|
||||
rm -rf /
|
||||
dd if=
|
||||
mkfs
|
||||
usermod
|
||||
passwd
|
||||
```
|
||||
|
||||
```javascript
|
||||
// In JavaScript/Node
|
||||
require('child_process').exec('sudo')
|
||||
fs.rmdirSync('/', { recursive: true })
|
||||
process.setuid(0)
|
||||
```
|
||||
|
||||
```python
|
||||
# In Python
|
||||
os.system('sudo')
|
||||
import subprocess
|
||||
subprocess.run(['sudo', ...])
|
||||
```
|
||||
|
||||
**Medium Risk Patterns:**
|
||||
```bash
|
||||
npm install
|
||||
git clone
|
||||
curl | bash
|
||||
apt-get install
|
||||
brew install
|
||||
pip install
|
||||
mkdir -p
|
||||
touch
|
||||
echo > file
|
||||
```
|
||||
|
||||
**Low Risk Patterns:**
|
||||
```bash
|
||||
cat file.txt
|
||||
grep pattern
|
||||
find . -name
|
||||
ls -la
|
||||
echo "message"
|
||||
```
|
||||
|
||||
### Step 3: Check Dependencies
|
||||
|
||||
Review plugin.json or README for dependencies:
|
||||
|
||||
**Low Risk:**
|
||||
- No external dependencies
|
||||
- Pure JavaScript/Python/Ruby standard library
|
||||
- Read-only CLI tools (cat, grep, jq for reading only)
|
||||
|
||||
**Medium Risk:**
|
||||
- NPM packages listed
|
||||
- Python packages (via requirements.txt)
|
||||
- Common CLI tools (git, curl, wget)
|
||||
- Database connections (read/write)
|
||||
|
||||
**High Risk:**
|
||||
- System packages (apt, yum, brew)
|
||||
- Kernel modules
|
||||
- Root-level dependencies
|
||||
- Unsigned binaries
|
||||
- External scripts from unknown sources
|
||||
|
||||
### Step 4: Review File Operations
|
||||
|
||||
Check what directories the skill accesses:
|
||||
|
||||
**Low Risk:**
|
||||
- Reads from current directory only
|
||||
- Reads from specified input files
|
||||
- Writes reports to current directory
|
||||
|
||||
**Medium Risk:**
|
||||
- Reads/writes to ~/.claude/
|
||||
- Reads/writes to /tmp/
|
||||
- Creates files in user directories
|
||||
- Modifies project files
|
||||
|
||||
**High Risk:**
|
||||
- Accesses /etc/
|
||||
- Accesses /usr/ or /usr/local/
|
||||
- Accesses /sys/ or /proc/
|
||||
- Modifies system binaries
|
||||
- Accesses /var/log/
|
||||
|
||||
### Step 5: Network Activity Assessment
|
||||
|
||||
**Low Risk:**
|
||||
- No network activity
|
||||
- Reads from local cache only
|
||||
|
||||
**Medium Risk:**
|
||||
- HTTP GET requests to public APIs
|
||||
- Documented API endpoints
|
||||
- Read-only data fetching
|
||||
- HTTPS only
|
||||
|
||||
**High Risk:**
|
||||
- HTTP POST with sensitive data
|
||||
- Unclear network destinations
|
||||
- Raw socket operations
|
||||
- Arbitrary URL from user input
|
||||
- Self-updating mechanism
|
||||
|
||||
## Automatic Risk Scoring
|
||||
|
||||
Use this scoring system:
|
||||
|
||||
```javascript
|
||||
function assessSkillRisk(skill) {
|
||||
let score = 0;
|
||||
|
||||
// File operations
|
||||
if (mentions(skill, "read", "parse", "analyze")) score += 1;
|
||||
if (mentions(skill, "write", "create", "modify")) score += 3;
|
||||
if (mentions(skill, "delete", "remove", "rm -rf")) score += 8;
|
||||
|
||||
// System operations
|
||||
if (mentions(skill, "npm install", "pip install")) score += 3;
|
||||
if (mentions(skill, "apt-get", "brew install")) score += 5;
|
||||
if (mentions(skill, "sudo", "systemctl", "service")) score += 10;
|
||||
|
||||
// File paths
|
||||
if (accesses(skill, "~/", "/tmp/")) score += 2;
|
||||
if (accesses(skill, "/etc/", "/usr/")) score += 8;
|
||||
|
||||
// Network
|
||||
if (mentions(skill, "fetch", "API", "curl")) score += 2;
|
||||
if (mentions(skill, "download", "wget")) score += 3;
|
||||
|
||||
// Process operations
|
||||
if (mentions(skill, "exec", "spawn", "child_process")) score += 4;
|
||||
|
||||
// Determine risk level
|
||||
if (score <= 3) return "low"; // Worktree
|
||||
if (score <= 10) return "medium"; // Docker
|
||||
return "high"; // VM
|
||||
}
|
||||
```
|
||||
|
||||
**Scoring Reference:**
|
||||
- 0-3: Low Risk → Git Worktree
|
||||
- 4-10: Medium Risk → Docker
|
||||
- 11+: High Risk → VM
|
||||
|
||||
## Special Cases
|
||||
|
||||
### Unknown or Unreviewed Skills
|
||||
|
||||
**Default:** High Risk (VM isolation)
|
||||
|
||||
Even if skill appears low risk, use VM for first test of:
|
||||
- Skills from external repositories
|
||||
- Skills without documentation
|
||||
- Skills with obfuscated code
|
||||
- Skills from untrusted authors
|
||||
|
||||
### Skills in Active Development
|
||||
|
||||
**Recommendation:** Medium Risk (Docker)
|
||||
|
||||
For your own skills during development:
|
||||
- Start with Git Worktree for speed
|
||||
- Use Docker before committing
|
||||
- Use VM before public release
|
||||
|
||||
### Skills from Marketplace
|
||||
|
||||
**Recommendation:** Follow listed risk level
|
||||
|
||||
Trusted marketplace skills can use their documented risk level.
|
||||
|
||||
## Override Cases
|
||||
|
||||
User can always override automatic detection:
|
||||
|
||||
```
|
||||
test skill low-risk-skill in vm # More isolation than needed (safe but slow)
|
||||
test skill high-risk-skill in docker # Less isolation (not recommended)
|
||||
```
|
||||
|
||||
**Warn user if choosing lower isolation than recommended.**
|
||||
|
||||
## Risk Re-assessment
|
||||
|
||||
Re-assess risk if skill is updated:
|
||||
- Major version changes
|
||||
- New dependencies added
|
||||
- New file operations
|
||||
- Expanded scope
|
||||
|
||||
## Decision Tree
|
||||
|
||||
```
|
||||
Start
|
||||
|
|
||||
├─ Does skill read files only?
|
||||
| └─ YES → Low Risk (Worktree)
|
||||
| └─ NO → Continue
|
||||
|
|
||||
├─ Does skill install packages or modify files?
|
||||
| └─ YES → Medium Risk (Docker)
|
||||
| └─ NO → Continue
|
||||
|
|
||||
├─ Does skill modify system configs or use sudo?
|
||||
| └─ YES → High Risk (VM)
|
||||
| └─ NO → Continue
|
||||
|
|
||||
└─ Is skill from untrusted source?
|
||||
└─ YES → High Risk (VM)
|
||||
└─ NO → Medium Risk (Docker)
|
||||
```
|
||||
|
||||
## Example Assessments
|
||||
|
||||
### Example 1: "code-formatter"
|
||||
|
||||
**Description:** Formats JavaScript/TypeScript files using prettier
|
||||
|
||||
**Analysis:**
|
||||
- Reads files: Yes (score: +1)
|
||||
- Writes files: Yes (score: +3)
|
||||
- System commands: No
|
||||
- Dependencies: prettier (npm package) (score: +3)
|
||||
- File paths: Current directory only
|
||||
|
||||
**Total Score:** 7
|
||||
**Risk Level:** Medium → Docker
|
||||
|
||||
**Reasoning:** Modifies files but limited to project directory. Docker provides adequate isolation.
|
||||
|
||||
### Example 2: "log-analyzer"
|
||||
|
||||
**Description:** Parses log files and generates HTML report
|
||||
|
||||
**Analysis:**
|
||||
- Reads files: Yes (score: +1)
|
||||
- Writes files: Yes (HTML report) (score: +3)
|
||||
- System commands: No
|
||||
- Dependencies: None
|
||||
- File paths: Current directory + /tmp for temp files (score: +2)
|
||||
|
||||
**Total Score:** 6
|
||||
**Risk Level:** Medium → Docker
|
||||
|
||||
**Reasoning:** Safe operations but creates files. Docker ensures clean testing.
|
||||
|
||||
### Example 3: "system-auditor"
|
||||
|
||||
**Description:** Audits system security configuration
|
||||
|
||||
**Analysis:**
|
||||
- Reads files: Yes, including /etc/ (score: +1 + 8)
|
||||
- System commands: Runs systemctl, checks services (score: +10)
|
||||
- Dependencies: System tools
|
||||
- File paths: /etc/, /var/log/ (score: +8)
|
||||
|
||||
**Total Score:** 27
|
||||
**Risk Level:** High → VM
|
||||
|
||||
**Reasoning:** Accesses sensitive system directories and uses system commands. VM required.
|
||||
|
||||
### Example 4: "markdown-linter"
|
||||
|
||||
**Description:** Checks markdown files for style violations
|
||||
|
||||
**Analysis:**
|
||||
- Reads files: Yes (score: +1)
|
||||
- Writes files: No (only stdout)
|
||||
- System commands: No
|
||||
- Dependencies: None
|
||||
- File paths: Current directory only
|
||||
|
||||
**Total Score:** 1
|
||||
**Risk Level:** Low → Git Worktree
|
||||
|
||||
**Reasoning:** Pure read-only analysis. Worktree is sufficient and fast.
|
||||
|
||||
---
|
||||
|
||||
**Remember:** When in doubt, choose higher isolation. It's better to be safe than to clean up a compromised system. Speed is secondary to security.
|
||||
543
skills/meta/skill-isolation-tester/data/side-effect-checklist.md
Normal file
543
skills/meta/skill-isolation-tester/data/side-effect-checklist.md
Normal file
@@ -0,0 +1,543 @@
|
||||
# Side Effect Detection Checklist
|
||||
|
||||
## Overview
|
||||
|
||||
This checklist helps identify all side effects caused by skill execution. Side effects are any changes to the system state beyond the skill's primary output. Proper detection ensures skills are well-behaved and clean up after themselves.
|
||||
|
||||
## Why Side Effects Matter
|
||||
|
||||
**Portability:** Skills with untracked side effects may not work for other users
|
||||
|
||||
**Cleanliness:** Leftover files and processes waste resources
|
||||
|
||||
**Security:** Unexpected system modifications are security risks
|
||||
|
||||
**Documentation:** Users need to know what a skill changes
|
||||
|
||||
## Categories of Side Effects
|
||||
|
||||
## 1. Filesystem Changes
|
||||
|
||||
### Files Created
|
||||
|
||||
**What to Check:**
|
||||
- Files in skill directory
|
||||
- Files in /tmp/ or /var/tmp/
|
||||
- Files in user home directory (~/)
|
||||
- Files in system directories (/usr/local/, /opt/)
|
||||
- Hidden files (.*) and cache directories (.cache/)
|
||||
- Lock files (.lock, .pid)
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Before execution
|
||||
find /path -type f > /tmp/before-files.txt
|
||||
|
||||
# After execution
|
||||
find /path -type f > /tmp/after-files.txt
|
||||
|
||||
# Compare
|
||||
diff /tmp/before-files.txt /tmp/after-files.txt | grep "^>" | sed 's/^> //'
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Temporary files in /tmp cleaned up before exit
|
||||
- ✅ Output files in current directory or specified location
|
||||
- ✅ Cache files in ~/.cache/skill-name/ (acceptable)
|
||||
- ❌ Random files scattered across filesystem
|
||||
- ❌ Files in system directories without explicit permission
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Cache files in proper location
|
||||
- **MEDIUM**: Temp files not cleaned up
|
||||
- **HIGH**: Files in system directories
|
||||
- **CRITICAL**: Files overwriting existing user data
|
||||
|
||||
### Files Modified
|
||||
|
||||
**What to Check:**
|
||||
- Project files (package.json, tsconfig.json, etc.)
|
||||
- Configuration files (.env, .config/)
|
||||
- System configs (/etc/*)
|
||||
- User configs (~/.bashrc, ~/.zshrc)
|
||||
- Git repository files (.git/)
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Take checksums before
|
||||
find /path -type f -exec md5sum {} \; > /tmp/before-checksums.txt
|
||||
|
||||
# After execution
|
||||
find /path -type f -exec md5sum {} \; > /tmp/after-checksums.txt
|
||||
|
||||
# Find modified files
|
||||
diff /tmp/before-checksums.txt /tmp/after-checksums.txt
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Only files explicitly in skill's scope modified
|
||||
- ✅ Backup created before modifying important files
|
||||
- ✅ Modifications clearly documented in output
|
||||
- ❌ Configuration files modified without notice
|
||||
- ❌ Git repository modified unexpectedly
|
||||
- ❌ System files changed
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Intended file modifications (skill's purpose)
|
||||
- **MEDIUM**: Unintended project file changes
|
||||
- **HIGH**: User config modifications without consent
|
||||
- **CRITICAL**: System file modifications
|
||||
|
||||
### Files Deleted
|
||||
|
||||
**What to Check:**
|
||||
- Files in skill scope (expected deletions)
|
||||
- Temp files created by skill
|
||||
- User files outside skill scope
|
||||
- System files
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Compare before/after file lists
|
||||
diff /tmp/before-files.txt /tmp/after-files.txt | grep "^<" | sed 's/^< //'
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Only temporary files created by skill deleted
|
||||
- ✅ Deletions are part of skill's documented purpose
|
||||
- ❌ User files deleted without explicit permission
|
||||
- ❌ Project files deleted accidentally
|
||||
- ❌ System files deleted
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Skill's own temp files deleted (cleanup)
|
||||
- **MEDIUM**: Unexpected file deletions in project
|
||||
- **HIGH**: User files deleted
|
||||
- **CRITICAL**: System files or important data deleted
|
||||
|
||||
### Directory Changes
|
||||
|
||||
**What to Check:**
|
||||
- New directories created
|
||||
- Working directory changed
|
||||
- Directories removed
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# List directories before/after
|
||||
find /path -type d > /tmp/before-dirs.txt
|
||||
find /path -type d > /tmp/after-dirs.txt
|
||||
diff /tmp/before-dirs.txt /tmp/after-dirs.txt
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Directories created for skill output
|
||||
- ✅ Temp directories in /tmp
|
||||
- ✅ Working directory restored after operations
|
||||
- ❌ Empty directories left behind
|
||||
- ❌ Directories created in unexpected locations
|
||||
|
||||
## 2. Process Management
|
||||
|
||||
### Processes Created
|
||||
|
||||
**What to Check:**
|
||||
- Foreground processes (should complete)
|
||||
- Background processes (daemons, services)
|
||||
- Child processes (spawned by skill)
|
||||
- Zombie processes
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Before execution
|
||||
ps aux > /tmp/before-processes.txt
|
||||
|
||||
# After execution (wait 30 seconds)
|
||||
sleep 30
|
||||
ps aux > /tmp/after-processes.txt
|
||||
|
||||
# Find new processes
|
||||
diff /tmp/before-processes.txt /tmp/after-processes.txt | grep "^>"
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ All skill processes complete and exit
|
||||
- ✅ No orphaned child processes
|
||||
- ✅ Background services documented if needed
|
||||
- ❌ Processes still running after skill exits
|
||||
- ❌ Zombie processes
|
||||
- ❌ High CPU/memory usage processes
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Short-lived child processes that exit cleanly
|
||||
- **MEDIUM**: Background processes that should have been stopped
|
||||
- **HIGH**: Orphaned processes consuming resources
|
||||
- **CRITICAL**: Runaway processes (infinite loops, memory leaks)
|
||||
|
||||
### Process Resource Usage
|
||||
|
||||
**What to Check:**
|
||||
- CPU usage during and after execution
|
||||
- Memory consumption
|
||||
- Disk I/O
|
||||
- Network I/O
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Monitor during execution
|
||||
top -b -n 1 > /tmp/resource-usage.txt
|
||||
|
||||
# Or use htop, ps aux, etc.
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Reasonable resource usage for task
|
||||
- ✅ Resources released after completion
|
||||
- ❌ 100% CPU for extended time
|
||||
- ❌ Memory leaks (growing usage)
|
||||
- ❌ Excessive disk I/O
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Temporary spike during execution
|
||||
- **MEDIUM**: Higher than expected but acceptable
|
||||
- **HIGH**: Excessive usage (> 80% CPU, > 1GB RAM)
|
||||
- **CRITICAL**: Resource exhaustion (OOM, disk full)
|
||||
|
||||
## 3. System Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
**What to Check:**
|
||||
- New environment variables set
|
||||
- Modified PATH, HOME, etc.
|
||||
- Shell configuration changes
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Before
|
||||
env | sort > /tmp/before-env.txt
|
||||
|
||||
# After
|
||||
env | sort > /tmp/after-env.txt
|
||||
|
||||
# Compare
|
||||
diff /tmp/before-env.txt /tmp/after-env.txt
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ No permanent environment changes
|
||||
- ✅ Temporary env vars for skill only
|
||||
- ❌ PATH modified globally
|
||||
- ❌ System env vars changed
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Temporary env vars in skill scope
|
||||
- **MEDIUM**: PATH modified in current shell
|
||||
- **HIGH**: .bashrc/.zshrc modified
|
||||
- **CRITICAL**: System-wide env changes
|
||||
|
||||
### System Services
|
||||
|
||||
**What to Check:**
|
||||
- Systemd services started
|
||||
- Cron jobs created
|
||||
- Launch agents/daemons (macOS)
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Linux
|
||||
systemctl list-units --type=service > /tmp/before-services.txt
|
||||
# After
|
||||
systemctl list-units --type=service > /tmp/after-services.txt
|
||||
diff /tmp/before-services.txt /tmp/after-services.txt
|
||||
|
||||
# Cron jobs
|
||||
crontab -l > /tmp/before-cron.txt
|
||||
# After
|
||||
crontab -l > /tmp/after-cron.txt
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ No services unless explicitly documented
|
||||
- ✅ Services stopped after skill exits
|
||||
- ❌ Services left running
|
||||
- ❌ Cron jobs created without consent
|
||||
|
||||
**Severity:**
|
||||
- **MEDIUM**: Services that should have been stopped
|
||||
- **HIGH**: Unexpected service installations
|
||||
- **CRITICAL**: System services modified
|
||||
|
||||
### Package Installations
|
||||
|
||||
**What to Check:**
|
||||
- NPM packages (global)
|
||||
- Python packages (pip)
|
||||
- System packages (apt, brew)
|
||||
- Ruby gems, Go modules, etc.
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# NPM global packages
|
||||
npm list -g --depth=0 > /tmp/before-npm.txt
|
||||
# After
|
||||
npm list -g --depth=0 > /tmp/after-npm.txt
|
||||
diff /tmp/before-npm.txt /tmp/after-npm.txt
|
||||
|
||||
# System packages (Debian/Ubuntu)
|
||||
dpkg -l > /tmp/before-packages.txt
|
||||
# After
|
||||
dpkg -l > /tmp/after-packages.txt
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ All dependencies documented in README
|
||||
- ✅ Local installations (in project directory)
|
||||
- ❌ Global package installations without notice
|
||||
- ❌ System package changes
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Local project dependencies
|
||||
- **MEDIUM**: Global NPM packages (if documented)
|
||||
- **HIGH**: System packages installed
|
||||
- **CRITICAL**: Conflicting package versions
|
||||
|
||||
## 4. Network Activity
|
||||
|
||||
### Connections Established
|
||||
|
||||
**What to Check:**
|
||||
- HTTP/HTTPS requests
|
||||
- WebSocket connections
|
||||
- Database connections
|
||||
- SSH connections
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Monitor network during execution
|
||||
# macOS
|
||||
lsof -i -n -P | grep <skill-process>
|
||||
|
||||
# Linux
|
||||
netstat -tupn | grep <skill-process>
|
||||
|
||||
# Or use tcpdump, wireshark for detailed analysis
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ All network requests documented
|
||||
- ✅ HTTPS used for sensitive data
|
||||
- ✅ Connections properly closed
|
||||
- ❌ Unexpected outbound connections
|
||||
- ❌ Data sent to unknown servers
|
||||
- ❌ Connections left open
|
||||
|
||||
**Severity:**
|
||||
- **LOW**: Documented API calls (HTTPS)
|
||||
- **MEDIUM**: HTTP requests (not HTTPS)
|
||||
- **HIGH**: Unexpected network destinations
|
||||
- **CRITICAL**: Data exfiltration attempts
|
||||
|
||||
### Data Transmitted
|
||||
|
||||
**What to Check:**
|
||||
- API payloads
|
||||
- File uploads/downloads
|
||||
- Metrics/telemetry data
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Clear documentation of what's sent
|
||||
- ✅ User consent for data transmission
|
||||
- ✅ No sensitive data in plaintext
|
||||
- ❌ Telemetry without consent
|
||||
- ❌ Credentials sent over HTTP
|
||||
|
||||
## 5. Database & State
|
||||
|
||||
### Database Changes
|
||||
|
||||
**What to Check:**
|
||||
- Tables created/dropped
|
||||
- Records inserted/updated/deleted
|
||||
- Schema migrations
|
||||
- Indexes created
|
||||
|
||||
**How to Detect:**
|
||||
```sql
|
||||
-- Before (SQLite example)
|
||||
SELECT * FROM sqlite_master WHERE type='table';
|
||||
|
||||
-- After
|
||||
SELECT * FROM sqlite_master WHERE type='table';
|
||||
|
||||
-- Record counts
|
||||
SELECT COUNT(*) FROM each_table;
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Changes are part of skill's purpose
|
||||
- ✅ Backup created before modifications
|
||||
- ✅ Transactions used (rollback on error)
|
||||
- ❌ Unexpected table drops
|
||||
- ❌ Data loss without backup
|
||||
- ❌ Schema changes without migration docs
|
||||
|
||||
### Cache & Session State
|
||||
|
||||
**What to Check:**
|
||||
- Redis/Memcached keys
|
||||
- Session files
|
||||
- Browser storage (if skill uses web UI)
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Cache properly namespaced
|
||||
- ✅ Expired sessions cleaned up
|
||||
- ❌ Cache pollution
|
||||
- ❌ Stale session files
|
||||
|
||||
## 6. Permissions & Security
|
||||
|
||||
### File Permissions
|
||||
|
||||
**What to Check:**
|
||||
- File permission changes (chmod)
|
||||
- Ownership changes (chown)
|
||||
- ACL modifications
|
||||
|
||||
**How to Detect:**
|
||||
```bash
|
||||
# Before
|
||||
ls -la /path > /tmp/before-perms.txt
|
||||
|
||||
# After
|
||||
ls -la /path > /tmp/after-perms.txt
|
||||
diff /tmp/before-perms.txt /tmp/after-perms.txt
|
||||
```
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Appropriate permissions for created files
|
||||
- ✅ No overly permissive files (777)
|
||||
- ❌ Permissions changed on existing files
|
||||
- ❌ World-writable files created
|
||||
|
||||
**Severity:**
|
||||
- **MEDIUM**: Overly restrictive permissions
|
||||
- **HIGH**: Overly permissive permissions (777)
|
||||
- **CRITICAL**: System file permission changes
|
||||
|
||||
### Security Credentials
|
||||
|
||||
**What to Check:**
|
||||
- API keys in files or logs
|
||||
- Passwords in plaintext
|
||||
- Certificates/keys created
|
||||
- SSH keys modified
|
||||
|
||||
**Expected Behavior:**
|
||||
- ✅ Credentials stored securely (keychain, vault)
|
||||
- ✅ No credentials in logs or temp files
|
||||
- ❌ API keys in plaintext files
|
||||
- ❌ Passwords in shell history
|
||||
- ❌ Private keys with wrong permissions
|
||||
|
||||
**Severity:**
|
||||
- **HIGH**: Credentials in files
|
||||
- **CRITICAL**: Credentials exposed to other users
|
||||
|
||||
## Automated Detection Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# side-effect-detector.sh
|
||||
|
||||
BEFORE_DIR="/tmp/skill-test-before"
|
||||
AFTER_DIR="/tmp/skill-test-after"
|
||||
|
||||
mkdir -p "$BEFORE_DIR" "$AFTER_DIR"
|
||||
|
||||
# Capture before state
|
||||
capture_state() {
|
||||
local DIR="$1"
|
||||
find /tmp -type f > "$DIR/tmp-files.txt"
|
||||
ps aux > "$DIR/processes.txt"
|
||||
env | sort > "$DIR/env.txt"
|
||||
npm list -g --depth=0 > "$DIR/npm-global.txt" 2>/dev/null
|
||||
netstat -tupn > "$DIR/network.txt" 2>/dev/null
|
||||
# Add more as needed
|
||||
}
|
||||
|
||||
# Before
|
||||
capture_state "$BEFORE_DIR"
|
||||
|
||||
# Run skill
|
||||
echo "Execute skill now..."
|
||||
read -p "Press enter when skill completes..."
|
||||
|
||||
# After
|
||||
capture_state "$AFTER_DIR"
|
||||
|
||||
# Compare
|
||||
echo "=== Side Effects Detected ==="
|
||||
echo ""
|
||||
echo "Files in /tmp:"
|
||||
diff "$BEFORE_DIR/tmp-files.txt" "$AFTER_DIR/tmp-files.txt" | grep "^>" | wc -l
|
||||
|
||||
echo "Processes:"
|
||||
diff "$BEFORE_DIR/processes.txt" "$AFTER_DIR/processes.txt" | grep "^>" | head -5
|
||||
|
||||
echo "Environment variables:"
|
||||
diff "$BEFORE_DIR/env.txt" "$AFTER_DIR/env.txt"
|
||||
|
||||
echo "NPM global packages:"
|
||||
diff "$BEFORE_DIR/npm-global.txt" "$AFTER_DIR/npm-global.txt"
|
||||
|
||||
# Detailed reports
|
||||
echo ""
|
||||
echo "Full reports in: $BEFORE_DIR and $AFTER_DIR"
|
||||
```
|
||||
|
||||
## Reporting Template
|
||||
|
||||
```markdown
|
||||
## Side Effects Report
|
||||
|
||||
### Filesystem Changes
|
||||
- **Files Created**: X files
|
||||
- /tmp/skill-temp-123.log (5KB)
|
||||
- ~/.cache/skill-name/data.json (15KB)
|
||||
- **Files Modified**: Y files
|
||||
- package.json (version updated)
|
||||
- **Files Deleted**: Z files
|
||||
- /tmp/old-cache.json
|
||||
|
||||
### Process Management
|
||||
- **Processes Created**: N
|
||||
- **Orphaned Processes**: M (list if > 0)
|
||||
- **Resource Usage**: Peak 45% CPU, 128MB RAM
|
||||
|
||||
### System Configuration
|
||||
- **Env Vars Changed**: None
|
||||
- **Services Started**: None
|
||||
- **Packages Installed**: jq (1.6)
|
||||
|
||||
### Network Activity
|
||||
- **Connections**: 3 HTTPS requests to api.example.com
|
||||
- **Data Transmitted**: 1.2KB (API calls)
|
||||
|
||||
### Database Changes
|
||||
- **Tables**: 1 created (skill_cache)
|
||||
- **Records**: 15 inserted
|
||||
|
||||
### Security
|
||||
- **Permissions**: All files 644 (appropriate)
|
||||
- **Credentials**: No sensitive data detected
|
||||
|
||||
### Overall Assessment
|
||||
✅ Cleanup: Mostly clean (3 temp files remaining)
|
||||
⚠️ Documentation: Missing jq dependency in README
|
||||
✅ Security: No issues
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Remember:** The goal is not zero side effects (that's impossible for useful skills), but **documented, intentional, and cleaned-up** side effects. Every side effect should be either part of the skill's purpose or properly cleaned up on exit.
|
||||
Reference in New Issue
Block a user