Initial commit

skills/scope/references/quick-reference.md

@@ -0,0 +1,375 @@
+# Scope Quick Reference
+
+Condensed reference for common Scope operations and patterns.
+
+## Command Cheat Sheet
+
+```bash
+# Doctor
+scope doctor run                   # Run all checks
+scope doctor run --only group-name # Run specific group
+scope doctor run --fix=false       # Check only
+scope doctor run --no-cache        # Force re-check
+scope doctor list                  # List all checks
+
+# Analyze
+scope analyze logs file.log            # Check log for errors
+scope analyze command -- cmd args      # Check command output
+scope analyze --extra-config dir/ file # Use extra configs
+
+# Report
+scope report ./script.sh     # Run & report script
+scope report -- command args # Run & report command
+
+# Version
+scope version # Show version
+```
+
+## YAML Templates
+
+### Known Error Template
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: error-name
+  description: What this error means
+spec:
+  pattern: 'regex pattern'
+  help: |
+    Explanation and steps to fix:
+    1. First step
+    2. Second step
+  fix:
+    prompt:
+      text: Permission prompt
+    commands:
+      - fix-command
+```
+
+### Doctor Group Template
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: group-name
+  description: What this group does
+spec:
+  include: when-required
+  needs:
+    - dependency-1
+  actions:
+    - name: action-name
+      description: What this checks
+      required: true
+      check:
+        paths:
+          - 'file.txt'
+        commands:
+          - ./bin/check.sh
+      fix:
+        commands:
+          - ./bin/fix.sh
+        helpText: |
+          Help if fix fails
+```
+
+### Report Location Template
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeReportLocation
+metadata:
+  name: location-name
+spec:
+  destination:
+    local:
+      directory: /tmp/reports
+    # OR githubIssue:
+    #   owner: org
+    #   repo: repo
+    # OR rustyPaste:
+    #   url: http://localhost:8000
+  additionalData:
+    command1: command-to-run
+```
+
+## Regex Pattern Examples
+
+```yaml
+# Match specific version error
+pattern: "ruby [0-9]+\\.[0-9]+\\.[0-9]+ is not installed"
+
+# Match file not found
+pattern: "cannot load such file -- .*/([^/]+)\\.(rb|so)"
+
+# Match Docker daemon not running
+pattern: "\\.colima/[^/]+/docker\\.sock.*Is the docker daemon running\\?"
+
+# Match Git lock error
+pattern: "Unable to create '.*\\.git/refs/heads/.*\\.lock'"
+
+# Match DNS resolution failure
+pattern: "Could not resolve host: ([a-zA-Z0-9.-]+)"
+
+# Match permission denied
+pattern: "Permission denied.*(/[^:]+)"
+
+# Character classes
+[[:digit:]]   # 0-9
+[[:alpha:]]   # a-z, A-Z
+[[:alnum:]]   # alphanumeric
+[[:space:]]   # whitespace
+
+# Quantifiers
+*             # 0 or more
++             # 1 or more
+?             # 0 or 1
+{n,m}         # between n and m
+
+# Escaping
+\\.           # literal dot
+\\[           # literal [
+\\(           # literal (
+```
+
+## Check/Fix Script Template
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+ACTION="${1:-check}"
+
+check() {
+  # Exit 0 if OK, non-zero if needs fix
+  if [[ condition ]]; then
+    return 0
+  else
+    echo "Check failed: reason" >&2
+    return 1
+  fi
+}
+
+fix() {
+  # Perform fix, exit 0 on success
+  echo "Fixing..."
+  # commands
+  return 0
+}
+
+case "$ACTION" in
+  check) check ;;
+  fix) fix ;;
+  *)
+    echo "Usage: $0 [check|fix]" >&2
+    exit 1
+    ;;
+esac
+```
+
+## Common Patterns
+
+### Version Check
+
+```yaml
+- name: min-version
+  description: Check minimum version
+  check:
+    commands:
+      - test "$(tool --version | cut -d' ' -f2)" = "1.2.3"
+  fix:
+    helpText: Update tool via Managed Software Center
+```
+
+### File Exists
+
+```yaml
+- name: config-exists
+  description: Config file exists
+  check:
+    commands:
+      - test -f .config
+  fix:
+    commands:
+      - ./bin/create-config.sh
+```
+
+### Service Running
+
+```yaml
+- name: service-up
+  description: Service is running
+  check:
+    commands:
+      - pgrep -x service-name
+  fix:
+    commands:
+      - brew services restart service-name
+```
+
+### Dependencies Installed
+
+```yaml
+- name: deps
+  description: Dependencies installed
+  check:
+    paths:
+      - package.json
+      - yarn.lock
+  fix:
+    commands:
+      - yarn install
+```
+
+### Path-Based Auto-Run
+
+```yaml
+# Runs fix when file changes
+- name: auto-update
+  check:
+    paths:
+      - config.yaml
+      - '**/*.conf'
+  fix:
+    commands:
+      - ./bin/reload.sh
+```
+
+## Validation Workflow
+
+```bash
+# 1. Create error file
+mkdir -p {config-root}/known-errors/category
+cat > {config-root}/known-errors/category/error.yaml << 'EOF'
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: my-error
+spec:
+  pattern: "error pattern"
+  help: How to fix
+EOF
+
+# 2. Create test file with actual error
+cat > {config-root}/known-errors/category/error.txt << 'EOF'
+Actual error output goes here
+EOF
+
+# 3. Test pattern
+scope analyze logs \
+  --extra-config {config-root} \
+  {config-root}/known-errors/category/error.txt
+
+# 4. Validate schema (if available)
+jsonschema validate schema.json error.yaml
+```
+
+## File Organization
+
+```
+# Gusto shared configs
+{config-root}/
+├── application/              # App-level (ruby, node, db)
+├── environment/              # System-level (homebrew, git)
+├── known-errors/
+│   ├── docker/
+│   ├── ruby/
+│   ├── git/
+│   └── {category}/
+│       ├── error-name.yaml
+│       └── error-name.txt   # Test file
+└── reports/
+
+# Project-specific
+.scope/
+├── project-name.yaml         # Main orchestrator
+├── db.yaml                  # Database setup
+├── ruby.yaml                # Language setup
+└── bin/                     # Helper scripts
+    ├── check-*.sh
+    └── fix-*.sh
+```
+
+## Debugging Checklist
+
+### Known Error Not Matching
+
+- [ ] Test regex: `echo "error" | rg "pattern"`
+- [ ] Check escaping of special chars
+- [ ] Verify test file has actual error
+- [ ] Try broader pattern first
+
+### Doctor Always Runs
+
+- [ ] Check path globs match: `ls -la path/pattern`
+- [ ] Verify check command exits 0: `./bin/check.sh; echo $?`
+- [ ] Try `--no-cache`
+- [ ] Check script is executable: `ls -l script.sh`
+
+### Dependencies Not Working
+
+- [ ] Run `scope doctor list` - see order
+- [ ] Verify `needs` names match exactly
+- [ ] Check for circular deps
+- [ ] Test with `--only group-name`
+
+### Script Issues
+
+- [ ] Add `set -euo pipefail` to scripts
+- [ ] Check relative path has `./` prefix
+- [ ] Make executable: `chmod +x script.sh`
+- [ ] Test standalone: `./bin/script.sh check`
+
+## Testing Tips
+
+```bash
+# Test regex patterns
+echo "error text here" | rg "pattern"
+
+# Test check command
+./bin/check.sh check
+echo "Exit code: $?"
+
+# Test doctor group in isolation
+scope doctor run --only group-name --no-cache
+
+# See what would run
+scope doctor list | grep group-name
+
+# Test with extra config
+scope analyze --extra-config /path/to/config file.log
+
+# Validate YAML syntax
+yamllint file.yaml
+
+# Check file matching
+ls -la path/to/files/**/*.pattern
+```
+
+## Environment Variables
+
+```bash
+# Report authentication
+SCOPE_GH_TOKEN=ghp_xxx        # GitHub PAT
+SCOPE_GH_APP_ID=123           # GitHub App
+SCOPE_GH_APP_KEY=/path/to/key # GitHub App key
+
+# Template variables (in YAML)
+{{ working_dir }} # Current working directory
+```
+
+## Common Gotchas
+
+1. **Regex escaping**: Use `\\.` for literal dot, not `.`
+2. **Relative paths**: Must start with `./` (relative to YAML file)
+3. **Check exit codes**: 0 = pass, non-zero = needs fix
+4. **Cache persistence**: Use `--no-cache` when testing
+5. **Pattern specificity**: Too broad = false positives, too narrow = misses errors
+6. **Script permissions**: Must be executable (`chmod +x`)
+7. **YAML indentation**: Use 2 spaces, not tabs
+8. **Action order**: Actions run in order defined
+9. **Dependency order**: Use `scope doctor list` to verify
+10. **Help text**: Use `|` for multi-line strings in YAML

skills/scope/references/real-world-examples.md

@@ -0,0 +1,579 @@
+# Real-World Scope Examples
+
+Curated examples from production Scope configurations showing battle-tested patterns.
+
+## Known Errors
+
+### Docker: Colima Not Running
+
+**File**: `known-errors/docker/default-colima-not-running.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: default-colima-not-running
+  description: The default instance of Colima is not running
+spec:
+  pattern: '.colima/default/docker.sock. Is the docker daemon running?'
+  help: |
+    Colima is not running. Start it by:
+    1. `scope doctor run --only company/docker@v1`
+
+    If that doesn't resolve the issue, reach out to us at @team
+    in the #help-channel channel in Slack for help.
+  fix:
+    prompt:
+      text: Run scope doctor?
+    commands:
+      - scope doctor run --only company/docker@v1
+```
+
+**Key Patterns**:
+
+- Uses escaped dot in path pattern: `\\.colima/`
+- Provides clear escalation path
+- Fix delegates to doctor group for complex multi-step resolution
+- Includes Slack channel for human help
+
+### Ruby: Gem Missing File
+
+**File**: `known-errors/ruby/gem-missing-file.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: gem-missing-file
+  description: A gem source file is missing, and fails to be loaded
+spec:
+  pattern: "/lib/ruby/([[:digit:]]\\.[[:digit:]]\\.[[:digit:]]|gems)/.* `(require|require_relative)': cannot load such file --.*/lib/ruby/gems/.*(LoadError)"
+  help: |
+    A gem source file is missing and fails to be loaded. The cause of this is
+    unknown and still being investigated (TICKET-123).
+
+    The solution is to reinstall the gems to fix the missing file:
+    1. Run `bundle pristine`
+  fix:
+    prompt:
+      text: Run bundle pristine?
+    commands:
+      - bundle pristine
+```
+
+**Key Patterns**:
+
+- Complex regex with alternation: `([[:digit:]]\\.[[:digit:]]\\.[[:digit:]]|gems)`
+- Uses character classes: `[[:digit:]]`
+- Multiple escaped characters in paths
+- References tracking issue in help text
+- Simple, safe fix command
+
+### Git: Cannot Lock Ref
+
+**File**: `known-errors/git/cannot-lock-ref.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: cannot-lock-ref
+  description: Git cannot create lock file for ref
+spec:
+  pattern: "error: cannot lock ref '[^']+': Unable to create '[^']+\\.lock': File exists"
+  help: |
+    Another git process is running or crashed leaving a lock file.
+
+    To resolve:
+    1. Check for running git processes: `ps aux | grep git`
+    2. If none running, remove the lock file mentioned in the error
+    3. Example: `rm .git/refs/heads/branch-name.lock`
+  fix:
+    prompt:
+      text: This requires manual intervention. Proceed with caution?
+    commands:
+      - echo "Check for git processes: ps aux | grep git"
+      - echo "If safe, manually remove the .lock file mentioned above"
+```
+
+**Key Patterns**:
+
+- Uses character class negation: `[^']+` (anything except single quote)
+- Escaped special characters: `\\.lock`
+- Fix provides diagnostic commands rather than automated fix
+- Warns user about manual intervention
+
+### MySQL: Connection Refused
+
+**File**: `known-errors/mysql/trilogy-connection-refused.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: trilogy-connection-refused
+  description: MySQL connection refused, service may not be running
+spec:
+  pattern: "Trilogy::ConnectionRefusedError.*Connection refused - connect\\(2\\)"
+  help: |
+    MySQL/MariaDB is not running or not accepting connections.
+
+    To fix:
+    1. Check if service is running: `brew services list | grep mysql`
+    2. Start the service: `brew services start mysql@8.0`
+    3. Or run: `scope doctor run --only database`
+  fix:
+    prompt:
+      text: Attempt to start MySQL service?
+    commands:
+      - brew services restart mysql@8.0
+```
+
+**Key Patterns**:
+
+- Escaped parentheses in regex: `\\(2\\)`
+- Provides multiple resolution paths
+- Delegates to doctor group for comprehensive fix
+- Uses `restart` instead of `start` (idempotent)
+
+## Doctor Groups
+
+### Ruby Version Management
+
+**File**: `application/ruby-version.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: ruby-version
+  description: Set up Ruby with accurate version
+spec:
+  include: when-required
+  needs:
+    - ruby-manager
+  actions:
+    - name: .ruby-version
+      description: Verify a valid .ruby-version file is present.
+      check:
+        commands:
+          - test -s .ruby-version
+      fix:
+        helpText: |
+          The .ruby-version file must exist and not be blank.
+    - name: install
+      description: Ensures correct version of ruby is installed
+      check:
+        paths:
+          - '{{ working_dir }}/.ruby-version'
+      fix:
+        commands:
+          - ./bin/ruby-version.sh install
+    - name: verify
+      description: Verify the desired ruby version and current version are the same
+      check:
+        commands:
+          - ./bin/ruby-version.sh verify
+      fix:
+        helpText: |
+          Something went wrong.
+          The ruby version was installed, but is not the version available in your current shell.
+          See error messages above for additional details and possible solutions.
+```
+
+**Key Patterns**:
+
+- Multiple sequential actions building on each other
+- First action has no fix commands, only helpText (manual intervention)
+- Second action watches file changes with `paths`
+- Third action validates end state
+- Uses template variable: `{{ working_dir }}`
+- Delegates complex logic to external script
+
+### Colima (Docker) Setup
+
+**File**: `environment/colima.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: company/docker@v1
+  description: Colima
+spec:
+  include: when-required
+  needs:
+    - homebrew
+  actions:
+    - name: install
+      description: company-docker is installed
+      check:
+        commands:
+          - ./bin/colima.sh check install
+      fix:
+        commands:
+          - ./bin/colima.sh fix install
+    - name: profile
+      description: The gusto profile exists
+      check:
+        commands:
+          - ./bin/colima.sh check profile
+      fix:
+        commands:
+          - ./bin/colima.sh fix profile
+        helpText: |-
+          The ~/.colima/company.yaml file still doesn't exist after running `sudo config-management`.
+          Please contact #ops-channel in slack.
+    - name: running
+      description: service and vm are running
+      check:
+        commands:
+          - ./bin/colima.sh check running
+      fix:
+        commands:
+          - ./bin/colima.sh fix running
+        helpText: |-
+          We were unable to start the company-docker service and/or the colima vm.
+          Please review the logs.
+
+          tail "$(brew --prefix)/var/log/service.log"
+
+          If you are not able to resolve the issue,
+          please contact #help-channel in slack.
+    - name: docker-context
+      description: docker context is set to gusto
+      check:
+        commands:
+          - ./bin/colima.sh check context
+      fix:
+        commands:
+          - ./bin/colima.sh fix context
+    - name: default-service
+      description: The default colima brew service is stopped
+      required: false
+      check:
+        commands:
+          - ./bin/colima.sh check default-service
+      fix:
+        commands:
+          - ./bin/colima.sh fix default-service
+    - name: default-profile
+      description: The default colima profile is stopped
+      required: false
+      check:
+        commands:
+          - ./bin/colima.sh check default-profile
+      fix:
+        commands:
+          - ./bin/colima.sh fix default-profile
+```
+
+**Key Patterns**:
+
+- Versioned name: `company/docker@v1` (allows breaking changes)
+- All actions delegate to same script with subcommands
+- Mix of required and optional actions
+- Complex multi-step setup
+- Detailed helpText with log locations
+- Shell command expansion in helpText: `$(brew --prefix)`
+- Last two actions are optional cleanup (`required: false`)
+
+### Brewfile Package Management
+
+**File**: `application/brewfile.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: brewfile
+  description: Homebrew managed packages
+spec:
+  include: when-required
+  needs:
+    - github-cli
+    - homebrew
+  actions:
+    - name: brew-bundle
+      description: Install Homebrew packages from Brewfile
+      check:
+        commands:
+          - ./bin/brew-bundle.sh check
+      fix:
+        commands:
+          - ./bin/brew-bundle.sh fix
+        helpText: |
+          brew dependencies cannot be satisfied
+
+          Please review the output above for errors and possible solutions.
+          If you need assistance, please contact #help-channel in slack.
+```
+
+**Key Patterns**:
+
+- Multiple dependencies ensure prerequisites installed first
+- Single action with simple check/fix delegation
+- Generic helpText directs to previous output
+- Minimal but effective
+
+### Version Requirements Check
+
+**File**: `.scope/scope.yaml` (project-specific)
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: scope
+spec:
+  include: when-required
+  needs: []
+  actions:
+    - name: minimum-scope-version
+      description: Ensures we have at least the minimum version of scope installed
+      check:
+        commands:
+          - ./bin/check-scope-version.sh check scope 2024.2.68
+      fix:
+        helpText: |
+          You don't have the minimum version of scope installed.
+          Check the Managed Software Center for updates.
+
+          If that doesn't work, please contact #help-channel in slack.
+    - name: minimum-gusto-scope-version
+      description: Ensures we have at least the minimum version of scope installed
+      check:
+        commands:
+          - ./bin/check-scope-version.sh check gusto 2025.05.15.0001
+      fix:
+        helpText: |
+          You don't have the minimum version of scope_config installed.
+          Check the Managed Software Center for updates.
+
+          If that doesn't work, please contact #help-channel in slack.
+```
+
+**Key Patterns**:
+
+- No dependencies (runs first)
+- No automated fix (requires external tool)
+- Passes version as argument to script
+- Consistent helpText pattern across actions
+
+### Orchestrator Pattern
+
+**File**: `.scope/project.yaml` (project-specific)
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: project
+  description: Application setup
+spec:
+  needs:
+    - scope
+    - company/environment@v1
+    - brewfile
+    - company/ruby@v1
+    - company/javascript@v1
+    - gitconfig
+    - lefthook
+    - db
+    - rubymine
+    - ruby-next
+    - kafka
+  actions: []
+```
+
+**Key Patterns**:
+
+- No actions, only dependencies
+- Orchestrates entire setup in correct order
+- Acts as entrypoint for `scope doctor run`
+- Clear dependency chain
+
+## Report Location
+
+**File**: `reports/report-location.yaml`
+
+```yaml
+apiVersion: scope.github.com/v1alpha
+kind: ScopeReportLocation
+metadata:
+  name: local
+spec:
+  destination:
+    local:
+      directory: /tmp/scope-reports
+  additionalData:
+    pwd: pwd
+    username: id -un
+    ruby: which ruby
+    node: which node
+    nodeVersion: node -v
+    scopeVersion: scope version
+    configVersion: config-tool --version
+```
+
+**Key Patterns**:
+
+- Local filesystem destination (no auth required)
+- Captures environment context
+- Uses simple shell commands
+- Platform-specific command: `pkgutil` (macOS)
+- Mix of path commands (`which`) and version commands
+
+## Helper Script Examples
+
+### Check/Fix Pattern
+
+**Example**: `bin/ruby-version.sh`
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+ACTION="${1:-}"
+COMMAND="${2:-}"
+
+check_file_exists() {
+  test -s .ruby-version
+}
+
+install_version() {
+  local desired_version
+  desired_version=$(cat .ruby-version)
+
+  if mise which ruby &> /dev/null; then
+    echo "Ruby is already available"
+    return 0
+  fi
+
+  mise install ruby@"${desired_version}"
+}
+
+verify_version() {
+  local desired_version current_version
+  desired_version=$(cat .ruby-version)
+  current_version=$(ruby --version | awk '{print $2}')
+
+  if [[ "$desired_version" == "$current_version" ]]; then
+    return 0
+  else
+    echo "Desired: $desired_version, Current: $current_version" >&2
+    return 1
+  fi
+}
+
+case "$ACTION" in
+  check)
+    check_file_exists
+    ;;
+  install)
+    install_version
+    ;;
+  verify)
+    verify_version
+    ;;
+  *)
+    echo "Usage: $0 [check|install|verify]" >&2
+    exit 1
+    ;;
+esac
+```
+
+**Key Patterns**:
+
+- Supports multiple subcommands
+- Extracts values from files
+- Uses command substitution
+- Provides clear error messages to stderr
+- Returns appropriate exit codes
+
+### Version Comparison
+
+**Example**: `bin/check-scope-version.sh`
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+ACTION="${1:-}"
+PACKAGE="${2:-}"
+MIN_VERSION="${3:-}"
+
+check_version() {
+  local current_version
+
+  case "$PACKAGE" in
+    scope)
+      current_version=$(scope version 2>&1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -1)
+      ;;
+    gusto)
+      current_version=$(config-tool --version 2>&1 | awk '/version:/ {print $2}')
+      ;;
+    *)
+      echo "Unknown package: $PACKAGE" >&2
+      exit 1
+      ;;
+  esac
+
+  if [[ "$(printf '%s\n' "$MIN_VERSION" "$current_version" | sort -V | head -1)" == "$MIN_VERSION" ]]; then
+    echo "Version $current_version meets minimum $MIN_VERSION"
+    return 0
+  else
+    echo "Version $current_version does not meet minimum $MIN_VERSION" >&2
+    return 1
+  fi
+}
+
+case "$ACTION" in
+  check)
+    check_version
+    ;;
+  *)
+    echo "Usage: $0 check <package> <min-version>" >&2
+    exit 1
+    ;;
+esac
+```
+
+**Key Patterns**:
+
+- Semantic version comparison using `sort -V`
+- Multiple package sources
+- Regex extraction of version numbers
+- Parameter validation
+
+## Lessons from Production
+
+### What Works Well
+
+1. **Versioned group names** (`company/docker@v1`) allow non-breaking changes
+2. **Orchestrator groups** with no actions simplify complex setups
+3. **Optional actions** (`required: false`) for nice-to-haves
+4. **Delegating to scripts** keeps YAML simple, logic testable
+5. **Consistent naming** (category/tool pattern) aids discovery
+6. **Rich helpText** with log locations and Slack channels
+7. **Multiple fix strategies** in help text (auto, manual, escalate)
+
+### Common Pitfalls
+
+1. **Overly broad patterns** catch unrelated errors
+2. **Missing escaping** in regex patterns
+3. **Hardcoded paths** instead of variables
+4. **Complex logic in YAML** instead of scripts
+5. **Missing error messages** when checks fail
+6. **No test files** make pattern validation harder
+7. **Circular dependencies** between groups
+
+### Scale Insights
+
+At 70+ known errors and 30+ doctor groups:
+
+- Categorization prevents overwhelming users
+- Consistent patterns make contribution easier
+- Test files are essential for maintenance
+- Versioning enables evolution without breaking changes
+- Clear ownership (Slack channels) reduces support burden

skills/scope/references/testing-guide.md

@@ -0,0 +1,661 @@
+# Scope Testing Guide
+
+Comprehensive guide for testing Scope configurations before deployment.
+
+## Testing Philosophy
+
+1. **Test patterns in isolation** before adding to YAML
+2. **Validate schema** before testing functionality
+3. **Test with real errors** using `.txt` files
+4. **Test incrementally** (pattern → YAML → integration)
+5. **Automate regression tests** for known errors
+
+## Known Error Testing
+
+### 1. Test Regex Pattern in Isolation
+
+```bash
+# Test pattern matches expected text
+echo "error: cannot lock ref 'refs/heads/main'" | rg "cannot lock ref"
+# Output: error: cannot lock ref 'refs/heads/main'
+
+# Test pattern doesn't match unrelated text
+echo "error: something else" | rg "cannot lock ref"
+# Output: (nothing - no match)
+
+# Test with actual error file
+rg "pattern" path/to/error-output.log
+```
+
+**Common Issues**:
+
+- Pattern too broad: matches too many things
+- Pattern too specific: misses variations
+- Missing escapes: special chars break regex
+- Wrong quantifiers: `*` vs `+` vs `?`
+
+### 2. Create Test File
+
+```bash
+# Create directory
+mkdir -p {config-root}/known-errors/category
+
+# Create test file with ACTUAL error output
+cat > {config-root}/known-errors/category/error-name.txt << 'EOF'
+[Paste actual error output here]
+This should be the real error text that users see
+Including all the context around it
+EOF
+```
+
+**Best Practices**:
+
+- Use real error output, not synthetic examples
+- Include surrounding context (lines before/after)
+- Test multiple variations if error has variants
+- Keep file size reasonable (< 100 lines)
+
+### 3. Create YAML Definition
+
+```bash
+cat > {config-root}/known-errors/category/error-name.yaml << 'EOF'
+apiVersion: scope.github.com/v1alpha
+kind: ScopeKnownError
+metadata:
+  name: error-name
+  description: Brief description
+spec:
+  pattern: "your pattern here"
+  help: |
+    How to fix this error
+EOF
+```
+
+### 4. Test Pattern Matching
+
+```bash
+# Test that pattern matches the test file
+scope analyze logs \
+  --extra-config {config-root} \
+  {config-root}/known-errors/category/error-name.txt
+```
+
+**Expected Output** (pattern matches):
+
+```
+Known Error: error-name
+Brief description
+
+How to fix this error
+```
+
+**Expected Output** (pattern doesn't match):
+
+```
+No known errors found
+```
+
+### 5. Validate Schema
+
+```bash
+# Install validator
+brew install sourcemeta/apps/jsonschema
+
+# Get schema (one-time setup)
+curl -o /tmp/ScopeKnownError.json \
+  https://github.com/oscope-dev/scope/raw/main/scope/schema/v1alpha.com.github.scope.ScopeKnownError.json
+
+# Validate
+jsonschema validate \
+  /tmp/ScopeKnownError.json \
+  {config-root}/known-errors/category/error-name.yaml
+```
+
+**Expected Output** (valid):
+
+```
+ok: {config-root}/known-errors/category/error-name.yaml
+```
+
+### 6. Test in Real Scenario
+
+```bash
+# Run the actual failing command and pipe to scope
+failing-command 2>&1 | scope analyze command
+
+# Or analyze existing log file
+scope analyze logs /path/to/error.log
+```
+
+## Doctor Group Testing
+
+### 1. Test Check Script Standalone
+
+```bash
+# Create test script
+cat > .scope/bin/test-check.sh << 'EOF'
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Your check logic here
+if [[ -f .required-file ]]; then
+  echo "Check passed"
+  exit 0
+else
+  echo "Check failed: .required-file missing" >&2
+  exit 1
+fi
+EOF
+
+chmod +x .scope/bin/test-check.sh
+
+# Test success case
+touch .required-file
+./.scope/bin/test-check.sh
+echo "Exit code: $?"
+# Expected: Check passed, Exit code: 0
+
+# Test failure case
+rm .required-file
+./.scope/bin/test-check.sh
+echo "Exit code: $?"
+# Expected: Check failed: .required-file missing, Exit code: 1
+```
+
+### 2. Test Fix Script Standalone
+
+```bash
+# Create fix script
+cat > .scope/bin/test-fix.sh << 'EOF'
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "Creating .required-file..."
+touch .required-file
+echo "Done"
+EOF
+
+chmod +x .scope/bin/test-fix.sh
+
+# Test fix
+rm -f .required-file
+./.scope/bin/test-fix.sh
+ls -la .required-file
+# Expected: File created
+```
+
+### 3. Create Doctor Group YAML
+
+```bash
+cat > .scope/test-group.yaml << 'EOF'
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: test-group
+  description: Test group for validation
+spec:
+  include: when-required
+  needs: []
+  actions:
+    - name: test-action
+      description: Test action
+      check:
+        commands:
+          - ./bin/test-check.sh
+      fix:
+        commands:
+          - ./bin/test-fix.sh
+EOF
+```
+
+### 4. Verify Group is Detected
+
+```bash
+# List all groups
+scope doctor list | grep test-group
+
+# Expected output includes:
+# ScopeDoctorGroup/test-group    Test group for validation    .scope/test-group.yaml
+```
+
+### 5. Test Check-Only Mode
+
+```bash
+# Remove file to trigger check failure
+rm -f .required-file
+
+# Run in check-only mode
+scope doctor run --only test-group --fix=false
+
+# Expected: Shows check failed, but doesn't run fix
+```
+
+### 6. Test With Fix
+
+```bash
+# Remove file to trigger check failure
+rm -f .required-file
+
+# Run with fix enabled
+scope doctor run --only test-group
+
+# Expected: Check fails, fix runs, file created
+ls -la .required-file
+```
+
+### 7. Test Caching
+
+```bash
+# First run (should execute check)
+scope doctor run --only test-group
+# Expected: Runs check and fix if needed
+
+# Second run (should use cache if file-based)
+scope doctor run --only test-group
+# Expected: Skips if using path-based checks
+
+# Force re-run
+scope doctor run --only test-group --no-cache
+# Expected: Runs check again
+```
+
+### 8. Test Dependencies
+
+```bash
+# Create dependency group
+cat > .scope/dependency.yaml << 'EOF'
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: dependency
+spec:
+  include: when-required
+  needs: []
+  actions:
+    - name: setup
+      check:
+        commands:
+          - test -f .dependency-marker
+      fix:
+        commands:
+          - touch .dependency-marker
+EOF
+
+# Update test group to depend on it
+# Add to test-group.yaml:
+# needs:
+#   - dependency
+
+# Test dependency resolution
+rm -f .dependency-marker .required-file
+scope doctor run --only test-group
+
+# Expected: Runs dependency first, then test-group
+ls -la .dependency-marker .required-file
+```
+
+### 9. Validate Schema
+
+```bash
+# Get schema
+curl -o /tmp/ScopeDoctorGroup.json \
+  https://github.com/oscope-dev/scope/raw/main/scope/schema/v1alpha.com.github.scope.ScopeDoctorGroup.json
+
+# Validate
+jsonschema validate \
+  /tmp/ScopeDoctorGroup.json \
+  .scope/test-group.yaml
+```
+
+## Integration Testing
+
+### Test Complete Workflow
+
+```bash
+# 1. Clean slate
+rm -rf /tmp/scope-test
+mkdir -p /tmp/scope-test/.scope/bin
+
+# 2. Create complete setup
+cd /tmp/scope-test
+
+# Create check script
+cat > .scope/bin/check.sh << 'EOF'
+#!/usr/bin/env bash
+test -f .setup-complete
+EOF
+chmod +x .scope/bin/check.sh
+
+# Create fix script
+cat > .scope/bin/fix.sh << 'EOF'
+#!/usr/bin/env bash
+echo "Setting up..."
+sleep 1
+touch .setup-complete
+echo "Done"
+EOF
+chmod +x .scope/bin/fix.sh
+
+# Create doctor group
+cat > .scope/setup.yaml << 'EOF'
+apiVersion: scope.github.com/v1alpha
+kind: ScopeDoctorGroup
+metadata:
+  name: setup
+spec:
+  include: by-default
+  needs: []
+  actions:
+    - name: initialize
+      description: Initialize project
+      check:
+        commands:
+          - ./.scope/bin/check.sh
+      fix:
+        commands:
+          - ./.scope/bin/fix.sh
+        helpText: |
+          Setup failed. Try running manually:
+          ./.scope/bin/fix.sh
+EOF
+
+# 3. Test end-to-end
+scope doctor run
+
+# 4. Verify result
+test -f .setup-complete && echo "SUCCESS" || echo "FAILED"
+```
+
+## Regression Testing
+
+### Create Test Suite
+
+```bash
+# Create test runner
+cat > test-scope.sh << 'EOF'
+#!/usr/bin/env bash
+set -euo pipefail
+
+PASSED=0
+FAILED=0
+
+test_known_error() {
+  local name=$1
+  local yaml=$2
+  local test_file=$3
+
+  echo "Testing: $name"
+
+  if scope analyze logs --extra-config {config-root} "$test_file" | grep -q "$name"; then
+    echo "  ✓ Pattern matches"
+    ((PASSED++))
+  else
+    echo "  ✗ Pattern does not match"
+    ((FAILED++))
+  fi
+}
+
+# Test all known errors
+for yaml in {config-root}/known-errors/*/*.yaml; do
+  name=$(basename "$yaml" .yaml)
+  txt="${yaml%.yaml}.txt"
+
+  if [[ -f "$txt" ]]; then
+    test_known_error "$name" "$yaml" "$txt"
+  fi
+done
+
+echo ""
+echo "Results: $PASSED passed, $FAILED failed"
+
+exit $FAILED
+EOF
+
+chmod +x test-scope.sh
+
+# Run test suite
+./test-scope.sh
+```
+
+### Continuous Testing
+
+```bash
+# Watch for changes and re-test
+while true; do
+  inotifywait -r -e modify {config-root}/
+  ./test-scope.sh
+done
+```
+
+## Performance Testing
+
+### Measure Doctor Run Time
+
+```bash
+# Time complete run
+time scope doctor run
+
+# Time specific group
+time scope doctor run --only group-name
+
+# Compare cached vs uncached
+time scope doctor run            # With cache
+time scope doctor run --no-cache # Without cache
+```
+
+### Profile Cache Effectiveness
+
+```bash
+# First run (cold cache)
+scope doctor run > /tmp/run1.log 2>&1
+
+# Second run (warm cache)
+scope doctor run > /tmp/run2.log 2>&1
+
+# Compare
+diff /tmp/run1.log /tmp/run2.log
+```
+
+## Debugging Tests
+
+### Enable Verbose Output
+
+```bash
+# Scope doesn't have --verbose, but you can debug scripts
+# Add to scripts:
+set -x # Print commands before execution
+
+# Example:
+#!/usr/bin/env bash
+set -euxo pipefail # Added 'x' for debug output
+```
+
+### Capture Full Output
+
+```bash
+# Capture stdout and stderr
+scope doctor run --only group-name > /tmp/stdout.log 2> /tmp/stderr.log
+
+# Capture combined
+scope doctor run --only group-name &> /tmp/combined.log
+
+# Capture and display
+scope doctor run --only group-name 2>&1 | tee /tmp/output.log
+```
+
+### Test Specific Action
+
+```bash
+# Doctor groups run all actions in sequence
+# To test just one action, you can run the script directly:
+./.scope/bin/script.sh check
+./.scope/bin/script.sh fix
+```
+
+## Common Test Scenarios
+
+### Test: Pattern with Special Characters
+
+```bash
+# Pattern: "error: cannot lock ref 'refs/heads/main'"
+# Test: Does it escape properly?
+
+echo "error: cannot lock ref 'refs/heads/main'" > /tmp/test.txt
+
+# This should match:
+rg "cannot lock ref '[^']+'" /tmp/test.txt
+
+# This should NOT match (missing escape):
+rg "cannot lock ref 'refs/heads/main'" /tmp/test.txt # Literal string
+```
+
+### Test: Path-Based Check Triggers on Change
+
+```bash
+# Setup group with path check
+cat > .scope/path-test.yaml << 'EOF'
+spec:
+  actions:
+    - name: test
+      check:
+        paths:
+          - config.yaml
+      fix:
+        commands:
+          - echo "Config changed"
+EOF
+
+# First run - config doesn't exist, should trigger
+scope doctor run --only path-test
+
+# Create config
+echo "version: 1" > config.yaml
+
+# Second run - config exists, should trigger (first time seeing it)
+scope doctor run --only path-test
+
+# Third run - config unchanged, should NOT trigger
+scope doctor run --only path-test
+
+# Change config
+echo "version: 2" > config.yaml
+
+# Fourth run - config changed, should trigger
+scope doctor run --only path-test
+```
+
+### Test: Dependency Order
+
+```bash
+# Create test that verifies order
+cat > .scope/dep-order-test.yaml << 'EOF'
+spec:
+  needs:
+    - first
+    - second
+  actions:
+    - name: verify
+      check:
+        commands:
+          - test -f .first-ran
+          - test -f .second-ran
+EOF
+
+# Run and check log order
+scope doctor run --only dep-order-test 2>&1 | grep -E "(first|second|verify)"
+```
+
+## Best Practices
+
+### DO
+
+✓ Test regex patterns with `rg` before adding to YAML
+✓ Create `.txt` test files with real error output
+✓ Validate schema before testing functionality
+✓ Test scripts standalone before integrating
+✓ Use `--no-cache` when testing changes
+✓ Test both success and failure paths
+✓ Test dependency resolution
+✓ Keep test files small and focused
+✓ Use version control to track test changes
+
+### DON'T
+
+✗ Skip testing regex patterns in isolation
+✗ Use synthetic error examples
+✗ Assume patterns work without testing
+✗ Test only the happy path
+✗ Forget to make scripts executable
+✗ Hard-code paths in tests
+✗ Test in production first
+✗ Commit without validation
+
+## Troubleshooting Tests
+
+### Pattern doesn't match test file
+
+```bash
+# Debug steps:
+# 1. Check file encoding
+file -I test-file.txt
+
+# 2. Check for hidden characters
+cat -A test-file.txt
+
+# 3. Test pattern piece by piece
+rg "simple" test-file.txt
+rg "simple.*pattern" test-file.txt
+rg "full.*complex.*pattern" test-file.txt
+
+# 4. Check escaping
+rg "pattern with \. dot" test-file.txt
+```
+
+### Check always fails
+
+```bash
+# Debug steps:
+# 1. Run script manually
+./.scope/bin/check.sh
+echo "Exit code: $?"
+
+# 2. Check for syntax errors
+bash -n ./.scope/bin/check.sh
+
+# 3. Add debug output
+set -x in script
+
+# 4. Check permissions
+ls -l ./.scope/bin/check.sh
+```
+
+### Fix runs but doesn't work
+
+```bash
+# Debug steps:
+# 1. Check fix script exit code
+./.scope/bin/fix.sh
+echo "Exit code: $?"
+
+# 2. Check what fix actually does
+./.scope/bin/fix.sh
+ls -la # Check if files created
+
+# 3. Run check after fix
+./.scope/bin/fix.sh && ./.scope/bin/check.sh
+echo "Check exit code: $?"
+```
+
+### Cache causes issues
+
+```bash
+# Solutions:
+# 1. Always use --no-cache when testing
+scope doctor run --only test --no-cache
+
+# 2. Clear cache manually (implementation-specific)
+# Check scope docs for cache location
+
+# 3. Use command-based checks instead of path-based
+# Commands don't use cache
+```