zhongwei/gh-secondsky-sap-skills-skills-skill-review
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
gh-secondsky-sap-skills-ski…/SKILL.md
Zhongwei Li 1df0ef2fbe Initial commit
2025-11-30 08:55:41 +08:00

18 KiB
Raw Permalink Blame History

name: skill-review description: | Comprehensive deep-dive documentation review process for sap-skills repository. Use this skill when investigating suspected issues in a skill, major package version updates detected, skill last verified >3 months ago, before marketplace submission, or when examples seem outdated. Performs systematic 14-phase audit: pre-review setup, standards compliance (exact YAML validation rules), official docs verification (Context7/WebFetch), code examples audit, cross-file consistency, dependency version checks, progressive disclosure architecture review, conciseness & degrees of freedom audit, anti-pattern detection, testing & evaluation review, security & MCP considerations, issue categorization by severity, fix implementation with minimal user questions, and post-fix verification. Enforces official Claude best practices including name validation (64 chars, lowercase/hyphens), description limits (1024 chars), SKILL.md line count (<500 lines), third-person writing style, progressive disclosure depth (one level), and multi-model testing verification. Prevents outdated documentation, incorrect API patterns, contradictory examples, version drift, broken links, YAML errors, schema inconsistencies, missing dependencies, Windows paths, inconsistent terminology, and over-explained content. Combines automated technical checks (YAML syntax, package versions, link validation, TODO markers) with AI-powered verification. Auto-fixes unambiguous issues, asks user only for architectural decisions or breaking change confirmations. Outputs severity-classified issue list (🔴 Critical / 🟡 High / 🟠 Medium / 🟢 Low) with evidence citations (GitHub URLs, official docs, changelogs) and detailed remediation plan. Keywords: skill review, skill audit, documentation review, version check, api verification, package updates, skill maintenance, quality assurance, standards compliance, breaking changes, github verification, official docs check, context7 verification, skill-review, audit-skill, review-documentation, check-skill-currency, verify-skill-accuracy, outdated skill, skill needs update, verify api methods, package version drift, stale documentation, contradictory examples, broken links check, yaml frontmatter validation, skill discovery test, production testing, sap skill review license: GPL-3.0 metadata: version: 1.3.0 last_verified: 2025-11-21 token_savings: ~80% errors_prevented: 31+ official_docs: https://github.com/secondsky/sap-skills triggers: - "review this skill" - "review the X skill" - "audit the skill" - "check if X needs updates" - "is X skill current" - "verify X documentation" - "X skill seems outdated" allowed-tools: - Read - Bash - Glob - Grep - WebFetch - WebSearch - Edit - Write

Skill Review Skill

Overview

The skill-review skill provides a comprehensive, systematic process for auditing skills in the sap-skills repository. It combines automated technical validation with AI-powered verification to ensure skills remain accurate, current, and high-quality.

Use this skill when:

  • Investigating suspected issues in a skill
  • Major package version updates released (e.g., better-auth 1.x → 2.x)
  • Skill last verified >90 days ago
  • Before submitting skill to marketplace
  • User reports errors following skill instructions
  • Examples seem outdated or contradictory

Production evidence: Production-tested audit methodology on skill quality reviews.

Quick Start

Invoke via Slash Command

/review-skill <skill-name>

Example:

/review-skill better-auth

Invoke via Skill (Proactive)

When Claude notices potential issues, it can suggest:

User: "I'm having trouble with better-auth and D1"

Claude: "I notice the better-auth skill was last verified 6 months ago.
Would you like me to review it? Better-auth recently released v1.3
with D1 changes."

What This Skill Does

14-Phase Systematic Audit

  1. Pre-Review Setup (5-10 min)

    • Install skill locally: ./scripts/install-skill.sh <skill-name>
    • Check current version and last verified date
    • Test skill discovery

  2. Standards Compliance (10-15 min)

    • Validate YAML frontmatter with exact rules:
      • name: Max 64 chars, pattern ^[a-z0-9-]+$, NO "anthropic" or "claude"
      • description: Max 1024 chars, NO XML tags (<tag>), non-empty
      • license: Present and valid (MIT, Apache-2.0, etc.)
    • SKILL.md line count: Body should be <500 lines (optimal performance)
    • Check keyword comprehensiveness
    • Verify third-person description style (NOT "You should..." but "This skill should be used when...")
    • Ensure gerund form naming (e.g., "processing-pdfs" not "pdf-processor")
    • Ensure directory structure matches spec

  3. Official Documentation Verification (15-30 min)

    • Use Context7 MCP or WebFetch to verify API patterns
    • Check GitHub for recent updates and issues
    • Verify package versions against npm registry
    • Compare with production repositories

  4. Code Examples & Templates Audit (20-40 min)

    • Verify import statements exist in current packages
    • Check API method signatures match official docs
    • Ensure schema consistency across files
    • Test templates build and run

  5. Cross-File Consistency (15-25 min)

    • Compare SKILL.md vs README.md examples
    • Verify "Bundled Resources" section matches actual files
    • Ensure configuration examples consistent

  6. Dependencies & Versions (10-15 min)

    • Run ./scripts/check-versions.sh <skill-name>
    • Check for breaking changes in package updates
    • Verify "Last Verified" date is recent

  7. Progressive Disclosure Architecture Review (10-15 min)

    • Check reference depth: Resources should be ONE LEVEL DEEP from SKILL.md
    • Verify files >100 lines have Table of Contents
    • Assess 3-tier model compliance:
      • Level 1 (Metadata): Always in context (~100 tokens)
      • Level 2 (SKILL.md body): Loaded when triggered (<500 lines)
      • Level 3 (Bundled resources): On-demand loading
    • Flag deeply nested references (references → sub-references → ❌)

  8. Conciseness & Degrees of Freedom Audit (15-20 min)

    • Identify over-explained concepts (Claude already knows this)
    • Flag verbose sections that could be trimmed
    • Assess degrees of freedom appropriateness:
      • High freedom: Exploratory tasks, vague requirements
      • Medium freedom: Conventional solutions, some flexibility
      • Low freedom: Fragile tasks, exact patterns required
    • Check for defaults with escape hatches (not endless options)
    • Apply "context window is a public good" mindset
    • Verify consistent terminology (same concept = same words throughout)

  9. Anti-Pattern Detection (10-15 min)

    • ❌ Windows-style paths (C:\path\file → use forward slashes)
    • ❌ Inconsistent terminology (endpoint/URL/path mixed usage)
    • ❌ Time-sensitive information ("as of 2024" → use "old patterns" sections)
    • ❌ Too many options without defaults (decision paralysis)
    • ❌ Deeply nested references (>1 level)
    • ❌ Vague phrases without examples
    • ❌ Missing input/output examples for templates
    • ❌ No feedback loops in complex workflows

  10. Testing & Evaluation Review (10-15 min)

    • Check for at least 3 test scenarios/evaluations
    • Verify multi-model consideration (Haiku/Sonnet/Opus may need different detail)
    • Assess if skill solves real problems vs imagined ones
    • Check for iterative development evidence (Claude A creates, Claude B tests)
    • Verify production testing claims with evidence

  11. Security & MCP Considerations (5-10 min)

    • Flag external URL fetches (potential risks)
    • Check for skills from untrusted sources warnings
    • Verify MCP tool references are fully qualified (ServerName:tool_name)
    • Review script permissions and error handling
    • Check "solve, don't punt" pattern (explicit error handling, not silent failures)
    • Marketplace schema compliance: Only standard fields allowed (name, source, description, version, category, keywords, author, license, repository) - NO custom fields like lastVerified

  12. Issue Categorization (10-20 min)

    • Classify by severity: 🔴 Critical / 🟡 High / 🟠 Medium / 🟢 Low
    • Document with evidence (GitHub URL, docs link, npm changelog)

  13. Fix Implementation (30 min - 4 hours)

    • Auto-fix unambiguous issues
    • Ask user only for architectural decisions
    • Update all affected files consistently
    • Bump version if breaking changes

  14. Post-Fix Verification (10-15 min)

    • Test skill discovery
    • Verify templates work
    • Check no contradictions remain
    • Commit with detailed changelog

Automated Checks (via script)

The skill runs ./scripts/review-skill.sh <skill-name> which checks:

  • ✅ YAML frontmatter syntax and required fields
  • ✅ Package version currency (npm)
  • ✅ Broken links (HTTP status)
  • ✅ TODO markers in code
  • ✅ File organization (expected directories exist)
  • ✅ "Last Verified" date staleness

Manual Verification (AI-powered)

Claude performs:

  • 🔍 API method verification against official docs
  • 🔍 GitHub activity and issue checks
  • 🔍 Production repository comparisons
  • 🔍 Code example correctness
  • 🔍 Schema consistency validation

Process Workflow

Step 1: Run Automated Checks

./scripts/review-skill.sh <skill-name>

Interpret output to identify technical issues.

Step 2: Execute Manual Verification

For Phase 3: Official Documentation Verification:

  1. Use Context7 MCP (if available):

    Use Context7 to fetch: /websites/<package-docs>
Search for: [API method from skill]

  2. Or use WebFetch:

    Fetch: [https://<official-docs-url>](https://<official-docs-url>)
Verify: [specific patterns]

  3. Check GitHub:

    Visit: [https://github.com/<org>/<repo>/commits/main](https://github.com/<org>/<repo>/commits/main)
Check: Last commit, recent changes
Search issues: [keywords from skill]

  4. Find production examples:

    WebSearch: "<package> cloudflare production github"
Compare: Do real projects match our patterns?

For Phase 4: Code Examples Audit:

  • Verify all imports exist (check official docs)
  • Check API method signatures match
  • Ensure schema consistency across files
  • Test templates actually work

Step 3: Categorize Issues

🔴 CRITICAL - Breaks functionality:

  • Non-existent API methods/imports
  • Invalid configuration
  • Missing required dependencies

🟡 HIGH - Causes confusion:

  • Contradictory examples across files
  • Inconsistent patterns
  • Outdated major versions

🟠 MEDIUM - Reduces quality:

  • Stale minor versions (>90 days)
  • Missing documentation sections
  • Incomplete error lists

🟢 LOW - Polish issues:

  • Typos, formatting inconsistencies
  • Missing optional metadata

Step 4: Fix Issues

Auto-fix when:

  • ✅ Fix is unambiguous (correct import from docs)
  • ✅ Evidence is clear
  • ✅ No architectural impact

Ask user when:

  • ❓ Multiple valid approaches
  • ❓ Breaking change decision
  • ❓ Architectural choice

Format for questions:

I found [issue]. There are [N] approaches:

1. [Approach A] - [Pros/Cons]
2. [Approach B] - [Pros/Cons]

Recommendation: [Default based on evidence]

Which would you prefer?

Step 5: Version Bump Assessment

If breaking changes:

  • Major: v1.0.0 → v2.0.0 (API patterns change)
  • Minor: v1.0.0 → v1.1.0 (new features, backward compatible)
  • Patch: v1.0.0 → v1.0.1 (bug fixes only)

Step 6: Generate Audit Report

## Skill Review Report: <skill-name>

**Date**: YYYY-MM-DD
**Trigger**: [Why review performed]
**Time Spent**: [Duration]

### Findings

🔴 CRITICAL (N): [List with evidence]
🟡 HIGH (N): [List with evidence]
🟠 MEDIUM (N): [List with evidence]
🟢 LOW (N): [List with evidence]

### Remediation

**Files Modified**: [List]
**Version Update**: [old] → [new]
**Breaking Changes**: Yes/No

### Verification

✅ Discovery test passed
✅ Templates work
✅ Committed: [hash]

### Recommendation

[Final assessment]

Example: better-auth Audit

Findings

Issue #1: Non-existent d1Adapter 🔴 CRITICAL

Location: references/cloudflare-worker-example.ts:17

Problem: Imports d1Adapter from 'better-auth/adapters/d1' which doesn't exist

Evidence:

Fix: Replace with drizzleAdapter from 'better-auth/adapters/drizzle'

Result

  • Files deleted: 3 (obsolete patterns)
  • Files created: 3 (correct patterns)
  • Lines changed: +1,266 net
  • Version: v1.0.0 → v2.0.0
  • Time: 3.5 hours

Bundled Resources

This skill references:

  1. planning/SKILL_REVIEW_PROCESS.md - Complete 14-phase manual guide
  2. scripts/review-skill.sh - Automated validation script
  3. .claude/commands/review-skill.md - Slash command definition

When Claude Should Invoke This Skill

Proactive triggers:

  • User mentions skill seems outdated
  • Package major version mentioned
  • User reports errors following skill
  • Checking metadata shows >90 days since verification

Explicit triggers:

  • "review the X skill"
  • "audit better-auth skill"
  • "is cloudflare-worker-base up to date?"
  • "check if tailwind-v4-shadcn needs updating"

Token Efficiency

Without this skill: ~25,000 tokens

  • Trial-and-error verification
  • Repeated doc lookups
  • Inconsistent fixes across files
  • Missing evidence citations

With this skill: ~5,000 tokens

  • Systematic process
  • Clear decision trees
  • Evidence-based fixes
  • Comprehensive audit trail

Savings: ~80% (20,000 tokens)

Common Issues Prevented

Content & API Issues

  1. Fake API adapters - Non-existent imports
  2. Stale API methods - Changed signatures
  3. Schema inconsistency - Different table names
  4. Outdated scripts - Deprecated approaches
  5. Contradictory examples - Multiple conflicting patterns
  6. Incomplete bundled resources - Listed files don't exist

Structure & Standards Issues

  1. YAML errors - Invalid frontmatter syntax
  2. Name too long - Exceeds 64 char limit
  3. Description too long - Exceeds 1024 char limit
  4. Invalid name format - Not lowercase/hyphens only
  5. Reserved words - Contains "anthropic" or "claude"
  6. Second-person descriptions - "You should..." instead of "This skill should be used when..."
  7. SKILL.md too long - Body exceeds 500 lines (performance impact)

Architecture Issues

  1. Deeply nested references - More than one level deep from SKILL.md
  2. Missing table of contents - Files >100 lines without navigation
  3. Over-explained concepts - Claude already knows this content

Quality & Testing Issues

  1. Missing keywords - Poor discoverability
  2. Version drift - Packages >90 days old
  3. Broken links - 404 documentation URLs
  4. No test scenarios - Missing evaluation cases
  5. No multi-model consideration - Only tested with one model

Anti-Patterns

  1. Windows-style paths - Backslashes instead of forward slashes
  2. Inconsistent terminology - Same concept, different words
  3. Time-sensitive info - "As of 2024" instead of version-based
  4. Too many options - No defaults provided
  5. No feedback loops - Complex workflows without validation steps

Security & MCP Issues

  1. Unqualified MCP references - Missing ServerName:tool_name format
  2. Silent error handling - "Punt" instead of "solve"
  3. Unvalidated external URLs - Fetching from untrusted sources
  4. Missing permissions warnings - Scripts without clear scope
  5. Non-standard marketplace fields - Custom fields rejected by schema (e.g., lastVerified)

Best Practices

  1. Always cite sources - GitHub URL, docs link, npm changelog
  2. No assumptions - Verify against current official docs
  3. Be systematic - Follow all 14 phases
  4. Fix consistency - Update all files, not just one
  5. Document thoroughly - Detailed commit messages
  6. Test after fixes - Verify skill still works

Known Limitations

  • Link checking requires network access
  • Package version checks need npm installed
  • Context7 MCP availability varies by package
  • Production repo search may need GitHub API
  • Manual phases require human judgment

Version History

v1.3.0 (2025-11-21)

  • Adapted for sap-skills repository
  • Updated repository references
  • Generalized examples for SAP skill development

v1.2.0 (2025-11-16)

  • Added marketplace schema compliance check (no custom fields)
  • Errors prevented: 31+ (was 30+)

v1.1.0 (2025-11-16)

  • Enhanced with official Claude best practices documentation
  • 14-phase systematic audit process (was 9-phase)
  • Added exact YAML validation rules (name: 64 chars, description: 1024 chars)
  • Added SKILL.md line count check (<500 lines)
  • Added progressive disclosure architecture review
  • Added conciseness & degrees of freedom audit
  • Added anti-pattern detection (Windows paths, inconsistent terminology)
  • Added testing & evaluation review (multi-model, 3+ test scenarios)
  • Added security & MCP considerations

v1.0.0 (2025-11-08)

  • Initial release
  • 9-phase systematic audit process
  • Automated script + manual guide
  • Slash command + skill wrapper

Additional Resources

Last verified: 2025-11-16 | Version: 1.2.0

Reference in New Issue View Git Blame Copy Permalink