gh-princespaghetti-claude-marketplace-learnfrompast/skills/dependency-evaluator/SIGNAL_DETAILS.md

Evaluation Signal Details

This file provides deep guidance for each of the 10 evaluation signals used in dependency assessment. For each signal, you'll find what it measures, how to investigate it, how to interpret results, and what constitutes red vs. green flags.

Assessment Philosophy: Ecosystem-Relative Evaluation

Use comparative assessment rather than absolute thresholds. What's "normal" varies significantly by:

• Ecosystem: npm vs PyPI vs Cargo vs Go have different cultural norms
• Package type: Frameworks vs utilities vs libraries have different expectations
• Maturity: New packages vs mature stable packages have different activity patterns

Throughout this guide:

• Red/green flags are framed as comparisons to ecosystem norms
• Specific numbers provide context, not rigid cutoffs
• "Significantly below/above norm" means outlier for package category in its ecosystem
• Always compare package to similar packages in same ecosystem before scoring

See ECOSYSTEM_GUIDES.md for ecosystem-specific baselines and norms.

1. Activity and Maintenance Patterns

What This Signal Measures

The frequency and consistency of package updates, bug fixes, and maintainer responsiveness. Active maintenance indicates the package is being improved and issues are being addressed.

What to Check

• Commit history and release cadence
• Time since last release
• How quickly critical bugs and security issues get addressed
• Issue triage responsiveness
• Consistency of maintenance over time

Ecosystem-Relative Assessment

Compare package activity against ecosystem norms rather than absolute thresholds. What's "normal" varies significantly by language, package type, and maturity.

Release Cadence Comparison:

• Red flag: Release cadence significantly below ecosystem norm for similar packages
  • npm actively-developed packages: Most release monthly or more; quarterly is typical minimum
  • Rust crates: Bi-monthly to quarterly is common; annual can be acceptable for stable crates
  • Python packages: Monthly to quarterly for active development
  • Go modules: Quarterly common; infrequent releases normal due to stdlib-first culture
• Assessment: Is this package's release pattern an outlier for its category within its ecosystem?

Commit Activity Comparison:

• Red flag: Commit activity has ceased while similar packages maintain activity
  • Look at comparable packages in same ecosystem/category
  • Mature stable libraries may legitimately have low commit frequency
  • New/actively-developed tools should show regular activity
• Green flag: Inactivity with zero open security issues may indicate package is "done" (complete, not abandoned)
• Context: Protocol implementations, math utilities, stable APIs may need few updates

Issue Response Comparison:

• Red flag: Issue response time significantly slower than ecosystem norm
  • npm: Hours to days typical for popular packages; weeks acceptable
  • Smaller ecosystems: Days to weeks is normal
  • Compare: Are issues being triaged, or ignored completely?
• Critical: Unaddressed security issues override all activity metrics

Backlog Assessment:

• Red flag: Issue backlog growing while similar packages maintain healthy triage
  • npm popular packages: 20-50 open issues may be normal if being triaged
  • Smaller projects: 10+ untriaged issues concerning
  • Key: Are maintainers responding, even if not immediately fixing?

Red Flags (Ecosystem-Relative)

• Release cadence significantly below ecosystem median for package category
• Commit activity ceased while comparable packages remain active
• Issue response time far slower than ecosystem norm
• Growing backlog with zero maintainer engagement
• Unaddressed security issues (absolute red flag regardless of ecosystem)

Green Flags (Ecosystem-Relative)

• Release cadence at or above ecosystem median
• Commit activity appropriate for package maturity and ecosystem
• Issue triage responsiveness comparable to or better than ecosystem norm
• Active PR review and merging
• Security issues addressed promptly even if feature development is slow

Common False Positives

• Low activity in mature libraries: A date library or cryptography implementation that hasn't changed in years might be complete, not abandoned. Check if issues are triaged and security updates still happen.
• Seasonal patterns: Academic or side-project packages may have irregular but acceptable maintenance patterns
• Small scope packages: A package that does one thing well may legitimately need few updates

2. Security Posture

What This Signal Measures

How the project handles security vulnerabilities, whether it has established security practices, and its history of security issues.

What to Check

• Security policy existence (SECURITY.md)
• Vulnerability disclosure process
• History of security advisories and CVEs
• Response time to past vulnerabilities
• Automated security scanning (Dependabot, Snyk badges)
• Proactive security measures

How to Investigate

• Search for CVE history: "<package-name>" CVE
• Look for security badges in README (Snyk, Dependabot)
• Review GitHub Security tab
• Check OSV database: https://osv.dev
• Run ecosystem security tools (npm audit, etc.)

Red Flags

• No security policy or disclosure process documented
• Slow CVE response time (30+ days from disclosure to patch)
• Multiple unpatched vulnerabilities
• No security scanning in CI/CD
• History of severe vulnerabilities
• Dismissive attitude toward security reports

Green Flags

• Published SECURITY.md with clear reporting process
• Quick CVE patches (< 7 days for critical issues)
• Security scanning enabled (Dependabot, Snyk)
• Bug bounty program
• Security-focused documentation
• Proactive security audits

Common False Positives

• Old, fixed vulnerabilities: Past CVEs that were quickly patched show good response, not poor security
• Reported but not exploitable: Some CVE reports may be theoretical or non-exploitable in practice

3. Community Health

What This Signal Measures

The breadth and engagement of the project's community, contributor diversity, and the "bus factor" (what happens if the main maintainer leaves).

What to Check

• Contributor diversity (single maintainer vs. team)
• PR merge rates and issue response times
• Stack Overflow activity
• Community forum engagement
• Maintainer communication style
• Organizational backing

How to Interpret

• health_percentage (from GitHub API) > 70 is good; < 50 suggests missing community files
• Multiple contributors (not just 1-2) indicates healthier bus factor
• Issues with comments show maintainer engagement; many 0-comment issues is a red flag
• PRs merged within days/weeks is healthy; months suggests slow maintenance

Red Flags

• Single maintainer with no backup or succession plan
• PRs sitting for months unreviewed
• Hostile or dismissive responses to issues
• No community engagement (Discord, Slack, forums)
• Maintainer burnout signals
• All recent activity from a single contributor

Green Flags

• Multiple active maintainers (3+ regular contributors)
• PRs reviewed within days
• Active Discord/Slack/forum community
• "Good first issue" labels for newcomers
• Welcoming, constructive communication
• Clear governance model or code of conduct
• Corporate or foundation backing

Common False Positives

• Single maintainer: Many excellent packages have one dedicated maintainer. This is higher risk but not automatically disqualifying if the maintainer is responsive and the codebase is simple enough to fork.
• Low community activity for niche tools: Specialized packages may have small but high-quality communities

4. Documentation Quality

What This Signal Measures

How well the package is documented, including API references, usage examples, migration guides, and architectural decisions.

What to Check

• Comprehensive API documentation
• Migration guides between major versions
• Real-world usage examples that work
• Architectural decision records (ADRs)
• TypeScript types / type definitions
• Inline code documentation
• Getting started tutorials

Red Flags

• Minimal or outdated README
• No API reference documentation
• No migration guides for breaking changes
• Examples that don't work with current version
• Missing type definitions for TypeScript
• No explanation of key concepts
• Documentation and code out of sync

Green Flags

• Comprehensive documentation site (e.g., Docusaurus, MkDocs)
• Versioned documentation matching releases
• Clear upgrade guides with examples
• Working examples and tutorials
• Interactive playgrounds or demos
• Architecture diagrams
• Searchable API reference
• Contribution guidelines

Common False Positives

• Self-documenting APIs: Very simple, intuitive APIs may not need extensive docs
• Code-focused projects: Some low-level libraries may have minimal prose but excellent code comments

5. Dependency Footprint

What This Signal Measures

The size and complexity of the dependency tree, including transitive dependencies and overall bundle size impact.

What to Check

• Number of direct dependencies
• Number of transitive dependencies
• Total dependency tree depth
• Quality and maintenance of transitive dependencies
• Bundle size impact
• Presence of native/binary dependencies

Interpreting Dependency Trees (Ecosystem-Relative)

Compare dependency counts against ecosystem norms:

Total Count Assessment:

• npm: 20-50 transitive deps common; 100+ raises concerns; 200+ is extreme
• Python/PyPI: 10-30 transitive deps typical; 50+ concerning for utilities
• Rust/Cargo: 20-40 transitive deps common (proc-macros inflate counts); 80+ heavy
• Go: 5-20 deps typical (stdlib-first culture); 40+ unusual
• Key: Compare functionality complexity to dependency count—simple utility with ecosystem-high dep count is red flag

Duplicate Versions:

• Multiple versions of same package indicate potential conflicts
• More concerning in npm (version resolution complex) than Cargo (strict resolution)

Tree Depth:

• Deep nesting (5+ levels) harder to audit regardless of ecosystem
• Rust proc-macro deps often add depth without adding risk

Abandoned Transitive Dependencies:

• Assess transitive deps using same maintenance criteria as direct deps
• One abandoned transitive dep may not be blocker; many suggests poor dep hygiene

Bundle Size vs. Functionality:

• npm: Compare to similar packages—is this outlier for what it does?
• Rust: Compile-time deps don't affect binary size, only build time
• Assess: Does bundle size match functionality provided?

Red Flags (Ecosystem-Relative)

• Dependency count in top quartile for package's functionality and ecosystem
• Transitive dependencies with known vulnerabilities
• Bundle size significantly above ecosystem norm for similar functionality
• Multiple unmaintained transitive dependencies
• Conflicting dependency version requirements
• Native dependencies when ecosystem-standard pure implementation available

Green Flags (Ecosystem-Relative)

• Dependency count at or below ecosystem median for package type
• All dependencies well-maintained and reputable
• Tree-shakeable / modular imports (npm, modern JS)
• Native deps only when necessary for performance/functionality
• Flat, shallow dependency structure
• Dependencies regularly updated

Common False Positives

• Framework packages: Full frameworks (React, Vue, Angular) legitimately have more dependencies
• Native performance: Some packages legitimately need native bindings for performance

6. Production Adoption

What This Signal Measures

Real-world usage of the package in production environments, indicating battle-tested reliability and community trust.

What to Check

• Download statistics and trends
• GitHub "Used by" count (dependents)
• Notable companies/projects using it
• Tech blog case studies
• Production deployment mentions
• Community recommendations

How to Investigate

• Check weekly/monthly download counts (npm, PyPI, crates.io)
• Review GitHub dependents graph
• Search " production" in tech blogs
• Look for case studies from reputable companies
• Check framework/platform official recommendations

Red Flags

• High download counts but no visible production usage (bot inflation)
• Only tutorial/example usage, no production mentions
• Declining download trends over time
• No notable adopters despite being old
• All usage from forks or abandoned projects

Green Flags

• Used by large, reputable organizations
• Growing or stable download trends
• Featured in production case studies
• Part of major frameworks' recommended ecosystems
• Referenced in official platform documentation
• Active "Who's using this" list

Common False Positives

• New packages: Legitimately new packages may have low downloads but high quality
• Niche tools: Specialized packages may have low downloads but be essential for their domain
• Internal tooling: Some excellent packages are used primarily internally

7. License Compatibility

What This Signal Measures

Whether the package's license and its dependencies' licenses are compatible with your project's license and intended use.

What to Check

• Package license type (MIT, Apache-2.0, GPL, etc.)
• License compatibility with your project
• License stability (no recent unexpected changes)
• Transitive dependency licenses
• Patent grants (especially Apache-2.0)

Red Flags

• Copyleft licenses (GPL, AGPL) for proprietary projects
• No license specified (all rights reserved by default)
• Recent license changes without notice
• Conflicting transitive dependency licenses
• Licenses with advertising clauses
• Ambiguous or custom licenses

Green Flags

• Permissive licenses (MIT, Apache-2.0, BSD-3-Clause)
• Clear LICENSE file in repository
• Consistent licensing across all dependencies
• SPDX identifiers used
• Patent grants (Apache-2.0)
• Well-understood, OSI-approved licenses

Common False Positives

• GPL for standalone tools: GPL is fine for CLI tools and dev dependencies that don't link into your code
• Dual licensing: Some projects offer both commercial and open-source licenses

8. API Stability

What This Signal Measures

How frequently the API changes in breaking ways, adherence to semantic versioning, and the deprecation process.

What to Check

• Changelog for breaking changes
• Semantic versioning adherence
• Deprecation policy and process
• Frequency of breaking changes in minor versions
• Migration tooling (codemods) for major upgrades
• Version number progression

How to Investigate

• Review CHANGELOG.md or GitHub releases
• Check version history for breaking change patterns
• Look for semver violations (breaking changes in patches/minors)
• Check for deprecation warnings before removal

Red Flags

• Frequent breaking changes in minor/patch versions
• No changelog or release notes
• No deprecation warnings before API removal
• Stuck at 0.x version for years
• Breaking changes without major version bumps
• No migration guides for major versions

Green Flags

• Strict semantic versioning adherence
• Clear, multi-release deprecation cycle
• Stable API (1.x+ with rare breaking changes)
• Migration codemods for major upgrades
• Detailed changelogs with examples
• Beta/RC releases before major versions
• Long-term support (LTS) versions

Common False Positives

• Pre-1.0 experimentation: 0.x versions are expected to have breaking changes
• Rapid iteration by design: Some frameworks intentionally move fast and document it clearly

9. Bus Factor and Funding

What This Signal Measures

The sustainability of the project if key contributors leave, and whether there's financial support for ongoing maintenance.

What to Check

• Organizational backing (CNCF, Apache Foundation, company sponsorship)
• OpenCollective or GitHub Sponsors presence
• Corporate contributor presence
• Full-time vs. volunteer maintainers
• Succession planning
• Funding transparency

How to Investigate

• Check for sponsor badges in README
• Look for corporate affiliations in contributor profiles
• Search " funding" or " sponsor"
• Check foundation membership (Linux Foundation, Apache, etc.)
• Review OpenCollective or GitHub Sponsors pages

Red Flags

• Solo volunteer maintainer for critical infrastructure
• No funding mechanism or sponsorship
• Maintainer burnout signals in issues/discussions
• Company backing withdrawn recently
• Underfunded relative to usage scale
• No succession plan

Green Flags

• Foundation backing (Linux Foundation, Apache, CNCF)
• Active sponsorship program with multiple sponsors
• Corporate maintainers (paid full-time)
• Sustainable funding model
• Multiple organizations contributing
• Clear governance structure
• Successor maintainers identified

Common False Positives

• Passion projects: Some maintainers prefer unfunded projects and sustain them long-term
• Mature, low-maintenance tools: Stable packages may not need significant funding

10. Ecosystem Momentum

What This Signal Measures

Whether the technology and ecosystem around the package is growing, stable, or declining.

What to Check

• Is the ecosystem migrating to alternatives?
• Framework/platform official support and alignment
• Technology trend direction
• Competitor activity
• Conference talks and blog posts
• Job market demand

How to Investigate

• Search for ecosystem discussions and trends
• Check if framework docs recommend alternatives
• Review technology radar reports (ThoughtWorks, etc.)
• Monitor competitor package growth
• Check conference talk mentions

Red Flags

• Ecosystem actively migrating to alternatives
• Deprecated by the framework it supports
• Based on sunset technology (Flash, CoffeeScript)
• No mentions at recent conferences
• Declining search trends
• Framework removed official support

Green Flags

• Growing ecosystem adoption
• Aligned with platform direction and roadmap
• Active plugin/extension ecosystem
• Regular conference mentions
• Increasing search and job trends
• Framework official recommendation
• Standards body involvement

Common False Positives

• Stable, mature ecosystems: Not every package needs to be trendy; stability can be valuable
• Niche domains: Specialized tools may have small but stable ecosystems

General Interpretation Guidelines

Context Matters

Always adjust signal interpretation based on:

• Dependency criticality: Auth libraries need stricter standards than dev tools
• Project scale: Enterprise projects have lower risk tolerance
• Domain complexity: Cryptography packages need different evaluation than UI libraries
• Ecosystem norms: Rust culture emphasizes different values than npm culture

Weighted Scoring

Not all signals are equally important:

• Critical dependencies: Prioritize Security, Maintenance, Funding
• Standard dependencies: Balance all signals
• Dev dependencies: Prioritize Maintenance, API Stability

Blocker Override

Any critical red flag (supply chain risk, security exploitation, license violation) should result in AVOID recommendation regardless of other scores.

Evidence-Based Assessment

Always cite specific data:

• Version numbers and dates
• Actual download counts or GitHub stars
• Specific CVE numbers
• Named organizations using the package
• Measured bundle sizes

Nuanced Judgment

Avoid purely mechanical scoring:

• A 3/5 in one signal with concerning details may be worse than 2/5 with clear mitigation
• Consider trajectory: improving vs. declining
• Weight recent data more than historical