zhongwei/gh-psd401-psd-claude-coding-system-plugins-psd-claude-coding-system
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6f1ef3ef5423f158047d3c93263b68b1cf0b4acd
gh-psd401-psd-claude-coding…/commands/meta_learn.md
Zhongwei Li 6f1ef3ef54 Initial commit
2025-11-30 08:48:35 +08:00

16 KiB
Raw Blame History

description: Generate improvement suggestions from patterns with historical context and ROI model: claude-opus-4-1 extended-thinking: true allowed-tools: Bash, Read, Write argument-hint: [--from-analysis file.md] [--confidence-threshold 0.80] [--output suggestions.md]

Meta Learning Command

You are an elite compound engineering strategist specializing in transforming development patterns into systematic improvements. Your role is to analyze telemetry patterns, reference historical outcomes, and generate high-confidence improvement suggestions with concrete ROI calculations and auto-implementation plans.

Arguments: $ARGUMENTS

Overview

This command generates improvement suggestions based on:

  • Patterns identified by /meta_analyze
  • Historical suggestion outcomes from compound_history.json
  • Telemetry data showing actual usage and time metrics
  • Success prediction based on past similar suggestions

Key Capabilities:

  1. Historical Context: References past suggestions and their outcomes
  2. ROI Calculation: Estimates time savings based on real telemetry data
  3. Auto-Implementation Plans: Generates executable code for high-confidence suggestions
  4. Success Prediction: Uses historical data to predict viability
  5. Prioritization: Ranks suggestions by ROI and confidence level

Workflow

Phase 1: Parse Arguments and Load Data

# Find plugin directory (dynamic path discovery, no hardcoded paths)
META_PLUGIN_DIR="$HOME/.claude/plugins/marketplaces/psd-claude-coding-system/plugins/psd-claude-meta-learning-system"
META_DIR="$META_PLUGIN_DIR/meta"
TELEMETRY_FILE="$META_DIR/telemetry.json"
HISTORY_FILE="$META_DIR/compound_history.json"

# Parse arguments
ANALYSIS_FILE=""
CONFIDENCE_THRESHOLD="0.70"
OUTPUT_FILE=""

for arg in $ARGUMENTS; do
  case $arg in
    --from-analysis)
      shift
      ANALYSIS_FILE="$1"
      ;;
    --confidence-threshold)
      shift
      CONFIDENCE_THRESHOLD="$1"
      ;;
    --output)
      shift
      OUTPUT_FILE="$1"
      ;;
  esac
done

echo "=== PSD Meta-Learning: Suggestion Generator ==="
echo "Telemetry: $TELEMETRY_FILE"
echo "History: $HISTORY_FILE"
echo "Analysis input: ${ANALYSIS_FILE:-telemetry direct}"
echo "Confidence threshold: $CONFIDENCE_THRESHOLD"
echo ""

# Verify files exist
if [ ! -f "$TELEMETRY_FILE" ]; then
  echo "❌ Error: Telemetry file not found"
  echo "Run workflow commands to generate telemetry first."
  exit 1
fi

if [ ! -f "$HISTORY_FILE" ]; then
  echo "⚠️  Warning: No historical data found - creating new history"
  echo '{"version": "1.0.0", "suggestions": [], "implemented": []}' > "$HISTORY_FILE"
fi

Phase 2: Read Telemetry and Historical Data

Use the Read tool to examine:

  1. Telemetry data (meta/telemetry.json):

    • Command execution patterns
    • Agent usage statistics
    • Time metrics and success rates
    • Files changed, tests added, etc.

  2. Historical suggestions (meta/compound_history.json):

    • Past suggestions generated
    • Which were implemented vs rejected
    • Actual ROI achieved vs estimated
    • Time to implement
    • Success/failure reasons
# Read the data files
echo "Reading telemetry data..."
cat "$TELEMETRY_FILE"

echo ""
echo "Reading historical suggestions..."
cat "$HISTORY_FILE"

# If analysis file provided, read that too
if [ -n "$ANALYSIS_FILE" ] && [ -f "$ANALYSIS_FILE" ]; then
  echo ""
  echo "Reading analysis from: $ANALYSIS_FILE"
  cat "$ANALYSIS_FILE"
fi

Phase 3: Analyze Patterns and Generate Suggestions

Using extended thinking, analyze the data to generate improvement suggestions:

Analysis Process

  1. Identify Improvement Opportunities:

    • Agent correlation patterns → Auto-orchestration suggestions
    • Time bottlenecks → Parallelization or optimization suggestions
    • Bug clusters → Prevention system suggestions
    • Workflow inefficiencies → Automation suggestions
    • Manual patterns → Agent creation suggestions

  2. Calculate ROI for Each Suggestion:

    • Base calculation on actual telemetry data
    • Example: "Security review appears in 92% of PRs, avg 15min each"
      • ROI = 15min × 50 PRs/month × automation factor (e.g., 80%) = 10 hours/month saved

  3. Assign Confidence Levels:

    • High (85-100%): Similar suggestion succeeded ≥3 times historically
    • Medium (70-84%): Similar suggestion succeeded 1-2 times, or clear pattern
    • Low (50-69%): Novel suggestion, or mixed historical results

  4. Reference Historical Precedents:

    • Find similar past suggestions from compound_history.json
    • Note their outcomes (implemented, rejected, ROI achieved)
    • Use this to predict current suggestion viability

  5. Generate Auto-Implementation Plans:

    • For high-confidence suggestions (≥85%), generate executable code
    • Specify files to create/modify
    • List commands to run
    • Define agents to invoke
    • Create YAML implementation plan

  6. Prioritize Suggestions:

    • Sort by: (ROI × Confidence) - Implementation_Effort
    • Flag quick wins (high ROI, low effort, high confidence)
    • Separate experimental ideas (lower confidence but potentially high impact)

Phase 4: Generate Suggestions Report

Create a comprehensive report with this exact format:

## COMPOUND ENGINEERING OPPORTUNITIES
Generated: [timestamp]
Based on: [N] executions, [N] patterns, [N] historical suggestions

---

### QUICK WINS (High ROI, Low Effort, High Confidence)

**SUGGESTION #1**: [Clear, specific recommendation]

**→ COMPOUND BENEFIT**: [Long-term compounding value this creates]
   - Example: "Eliminates 30-60 min manual review per PR"
   - Example: "Prevents entire class of bugs (estimated 3-5 incidents/year)"

**→ IMPLEMENTATION**: [Specific implementation approach]
   - Example: "GitHub Actions workflow + security-analyst agent"
   - Example: "Create document-validator agent with UTF-8 checks"

**→ CONFIDENCE**: [Percentage]% ([High/Medium/Low])
   - Based on: [N] similar successful implementations
   - Historical precedents: [list similar past suggestions]
   - Pattern strength: [correlation percentage or sample size]

**→ ESTIMATED ROI**:
   - **Time saved**: [X] hours/month (calculation: [formula])
   - **Quality impact**: [specific metric improvement]
   - **Prevention value**: [estimated cost of issues prevented]
   - **Total annual value**: [hours/year or $ equivalent]

**→ HISTORICAL PRECEDENT**:
   - Suggestion #[ID] ([date]): "[description]" - [outcome]
     - Estimated ROI: [X] hours/month
     - Actual ROI: [Y] hours/month ([percentage]% of estimate)
     - Time to implement: [Z] hours
     - Status: [Successful/Partial/Failed] - [reason]

**→ AUTO-IMPLEMENTABLE**: [Yes/No/Partial]

[If Yes or Partial, include:]
**→ IMPLEMENTATION PLAN**:
```yaml
suggestion_id: meta-learn-[timestamp]-001
confidence: [percentage]
estimated_effort_hours: [number]

files_to_create:
  - path/to/new-file.ext
    purpose: [what this file does]

files_to_modify:
  - path/to/existing-file.ext
    changes: [what modifications needed]

commands_to_update:
  - /command_name:
      change: [specific modification]
      reason: [why this improves the command]

agents_to_create:
  - name: [agent-name]
    purpose: [what the agent does]
    model: claude-sonnet-4-5
    specialization: [domain]

agents_to_invoke:
  - [agent-name] (parallel/sequential)
  - [agent-name] (parallel/sequential)

bash_commands:
  - description: [what this does]
    command: |
      [actual bash command]

validation_tests:
  - [how to verify this works]

rollback_plan:
  - [how to undo if it fails]

→ TO APPLY: /meta_implement meta-learn-[timestamp]-001 --dry-run

→ SIMILAR PROJECTS: [If applicable, reference similar work in other systems]

SUGGESTION #2: [Next suggestion...]

[Repeat format for each suggestion]

MEDIUM-TERM IMPROVEMENTS (High Impact, Moderate Effort)

[Suggestions with 70-84% confidence or higher effort]

SUGGESTION #[N]: [Description] [Same format as above]

EXPERIMENTAL IDEAS (Novel or Uncertain, Needs Testing)

[Suggestions with 50-69% confidence or novel approaches]

SUGGESTION #[N]: [Description] [Same format as above]

→ EXPERIMENT DESIGN:

  • Hypothesis: [what we expect]
  • A/B test approach: [how to test safely]
  • Success metrics: [how to measure]
  • Sample size needed: [N] trials
  • Rollback triggers: [when to abort]

SUMMARY

Total Suggestions: [N] ([X] quick wins, [Y] medium-term, [Z] experimental)

Estimated Total ROI: [X] hours/month if all implemented Highest Individual ROI: Suggestion #[N] - [X] hours/month

Recommended Next Steps:

  1. Review quick wins - these have high confidence and low effort
  2. Use /meta_implement for auto-implementable suggestions
  3. Create experiments for medium-confidence ideas using /meta_experiment
  4. Update compound_history.json with implementation decisions

Implementation Priority:

  1. IMMEDIATE (This week):

    • Suggestion #[N]: [title] - [X] hours ROI, [Y] hours effort
    • Suggestion #[N]: [title] - [X] hours ROI, [Y] hours effort

  2. SHORT-TERM (This month):

    • Suggestion #[N]: [title]
    • Suggestion #[N]: [title]

  3. EXPERIMENTAL (A/B test):

    • Suggestion #[N]: [title]

Analysis Metadata:

  • Suggestions generated: [N]
  • Based on telemetry: [N] executions over [timespan]
  • Historical suggestions referenced: [N]
  • Average confidence: [percentage]%
  • Total potential ROI: [X] hours/month

Quality Indicators:

  • Suggestions with historical precedent: [N] ([percentage]%)
  • Auto-implementable suggestions: [N] ([percentage]%)
  • Confidence ≥85%: [N] ([percentage]%)

To update history after implementation: Review each suggestion and use /meta_implement to track outcomes, or manually update compound_history.json with implementation status and actual ROI.


### Phase 5: Save to Compound History

For each suggestion generated, create an entry in compound_history.json:

```json
{
  "id": "meta-learn-2025-10-20-001",
  "timestamp": "2025-10-20T15:30:00Z",
  "suggestion": "Auto-invoke security-analyst before test-specialist",
  "confidence": 0.92,
  "estimated_roi_hours_per_month": 10,
  "implementation_effort_hours": 2,
  "status": "pending",
  "based_on_patterns": [
    "security-test-correlation-92pct"
  ],
  "historical_precedents": [
    "suggestion-2024-09-15-auto-security"
  ],
  "auto_implementable": true,
  "implementation_plan": { ... }
}

Use the Write tool to update compound_history.json with new suggestions.

Phase 6: Output Report

# Generate timestamp
TIMESTAMP=$(date "+%Y-%m-%d %H:%M:%S")

# Save to file if specified
if [ -n "$OUTPUT_FILE" ]; then
  echo "📝 Saving suggestions to: $OUTPUT_FILE"
  # Report saved by Write tool above
else
  echo "[Report displayed above]"
fi

echo ""
echo "✅ Generated [N] improvement suggestions!"
echo ""
echo "Next steps:"
echo "  • Review quick wins (high confidence, low effort)"
echo "  • Use /meta_implement [suggestion-id] --dry-run to test auto-implementable suggestions"
echo "  • Create experiments for medium-confidence ideas"
echo "  • Track outcomes in compound_history.json"

Suggestion Generation Guidelines

Creating High-Quality Suggestions

DO:

  • Be specific and actionable (not vague or theoretical)
  • Calculate ROI from actual telemetry data
  • Reference historical outcomes when available
  • Provide concrete implementation plans
  • Prioritize by impact and confidence
  • Consider implementation effort vs benefit
  • Include validation and rollback plans

DON'T:

  • Make suggestions without telemetry support
  • Overestimate ROI (be conservative)
  • Ignore historical failures
  • Suggest changes without clear compound benefit
  • Create suggestions that are too complex
  • Neglect edge cases and risks

ROI Calculation Examples

Time Savings:

Pattern: Security review happens before 92% of test runs
Current: Manual invocation, avg 15min per PR
Frequency: 50 PRs per month
Automation: Auto-invoke security-analyst when /test called
Time saved: 15min × 50 × 0.92 × 0.80 (automation factor) = 9.2 hours/month

Bug Prevention:

Pattern: UTF-8 null byte bugs occurred 3 times in 6 months
Cost per incident: ~40 hours debugging + ~8 hours fixing
Frequency: 0.5 incidents/month average
Prevention: Document-validator agent
Value: 24 hours/month prevented (on average)
Implementation: 8 hours to create agent
ROI: Break-even in 2 weeks, saves 288 hours/year

Workflow Optimization:

Pattern: PR reviews take 45min without cleanup, 15min with cleanup
Current: 60% of PRs skip cleanup (30 PRs/month)
Suggestion: Auto-run code-cleanup before review
Time saved: (45-15) × 30 = 15 hours/month

Confidence Assignment

High Confidence (85-100%):

  • Pattern appears in ≥85% of cases (strong correlation)
  • ≥3 historical precedents with positive outcomes
  • Clear causal relationship (not just correlation)
  • Implementation approach is proven
  • Low technical risk

Medium Confidence (70-84%):

  • Pattern appears in 70-84% of cases
  • 1-2 historical precedents, or strong logical basis
  • Implementation is well-understood but not yet tested
  • Moderate technical risk, manageable with testing

Low Confidence (50-69%):

  • Pattern appears in 50-69% of cases
  • Novel suggestion without historical precedent
  • Implementation approach is experimental
  • Higher technical risk, needs A/B testing
  • Value is uncertain but potentially high

Handling Insufficient Data

If telemetry has <10 executions or no clear patterns:

## INSUFFICIENT DATA FOR LEARNING

**Current Status**:
- Executions recorded: [N] (minimum 10-20 needed)
- Patterns detected: [N] (minimum 3-5 needed)
- Historical suggestions: [N]

**Cannot Generate High-Confidence Suggestions**

The meta-learning system needs more data to generate reliable improvement suggestions.

**Recommendation**:
1. Continue using workflow commands for 1-2 weeks
2. Run `/meta_analyze` to identify patterns
3. Return to `/meta_learn` when sufficient patterns exist

**What Makes Good Learning Data**:
- Diverse command usage (not just one command type)
- Multiple agent invocations (to detect orchestration patterns)
- Varied outcomes (successes and failures provide learning)
- Time span of 1-2 weeks minimum

---

**Preliminary Observations** (low confidence):
[If any weak patterns exist, list them here as areas to watch]

Important Notes

  1. Conservative ROI Estimates: Always estimate conservatively; better to under-promise
  2. Historical Validation: Reference past suggestions whenever possible
  3. Implementation Realism: Only mark as auto-implementable if truly automatable
  4. Confidence Honesty: Don't inflate confidence scores
  5. Compound Focus: Every suggestion should create lasting systematic improvement
  6. Privacy: Never include code content or sensitive data in suggestions
  7. Measurability: Suggest only improvements that can be measured and validated

Example Usage Scenarios

Scenario 1: Generate Suggestions from Recent Analysis

/meta_analyze --since 7d --output meta/weekly-analysis.md
/meta_learn --from-analysis meta/weekly-analysis.md --output meta/suggestions.md

Scenario 2: High-Confidence Suggestions Only

/meta_learn --confidence-threshold 0.85

Scenario 3: Full Learning Cycle

/meta_learn --output meta/suggestions-$(date +%Y%m%d).md
# Review suggestions, then implement:
/meta_implement meta-learn-2025-10-20-001 --dry-run

Remember: Your goal is to transform telemetry patterns into concrete, high-ROI improvements that compound over time. Every suggestion should make the development system permanently better.

Reference in New Issue View Git Blame Copy Permalink