7.7 KiB
7.7 KiB
Agent Prompt Metrics
Version: 1.0 Purpose: Quantitative metrics for measuring agent prompt quality Framework: BAIME dual-layer value functions applied to agents
Core Metrics
1. Success Rate
Definition: Percentage of tasks completed correctly
Calculation:
Success Rate = correct_completions / total_tasks
Thresholds:
- ≥90%: Excellent (production-ready)
- 80-89%: Good (minor refinements needed)
- 60-79%: Fair (needs improvement)
- <60%: Poor (major issues)
Example:
Tasks: 20
Correct: 17
Partial: 2
Failed: 1
Success Rate = 17/20 = 85% (Good)
2. Quality Score
Definition: Average quality rating of agent outputs (1-5 scale)
Rating Criteria:
- 5: Perfect - Accurate, complete, well-structured
- 4: Good - Minor gaps, mostly complete
- 3: Fair - Acceptable but needs improvement
- 2: Poor - Significant issues
- 1: Failed - Incorrect or unusable
Thresholds:
- ≥4.5: Excellent
- 4.0-4.4: Good
- 3.5-3.9: Fair
- <3.5: Poor
Example:
Task 1: 5/5 (perfect)
Task 2: 4/5 (good)
Task 3: 5/5 (perfect)
...
Task 20: 4/5 (good)
Average: 4.35/5 (Good)
3. Efficiency
Definition: Time and token usage per task
Metrics:
Time Efficiency = avg_time_per_task
Token Efficiency = avg_tokens_per_task
Thresholds (vary by agent type):
- Explore agent: <3 min, <5k tokens
- Code generation: <5 min, <10k tokens
- Analysis: <10 min, <20k tokens
Example:
Tasks: 20
Total time: 56 min
Total tokens: 92k
Time Efficiency: 2.8 min/task ✅
Token Efficiency: 4.6k tokens/task ✅
4. Reliability
Definition: Consistency of agent performance
Calculation:
Reliability = 1 - (std_dev(success_rate) / mean(success_rate))
Thresholds:
- ≥0.90: Very reliable (consistent)
- 0.80-0.89: Reliable
- 0.70-0.79: Moderately reliable
- <0.70: Unreliable (erratic)
Example:
Batch 1: 85% success
Batch 2: 90% success
Batch 3: 87% success
Batch 4: 88% success
Mean: 87.5%
Std Dev: 2.08
Reliability: 1 - (2.08/87.5) = 0.976 (Very reliable)
Composite Metrics
V_instance (Agent Performance)
Formula:
V_instance = 0.40 × success_rate +
0.30 × (quality_score / 5) +
0.20 × efficiency_score +
0.10 × reliability
Where:
- success_rate ∈ [0, 1]
- quality_score ∈ [1, 5], normalized to [0, 1]
- efficiency_score = 1 - (actual_time / target_time), capped at [0, 1]
- reliability ∈ [0, 1]
Target: V_instance ≥ 0.80
Example:
Success Rate: 85% = 0.85
Quality Score: 4.2/5 = 0.84
Efficiency: 2.8 min / 3 min target = 1 - 0.93 = 0.07, but we want faster so: 1.0 (under budget)
Reliability: 0.976
V_instance = 0.40 × 0.85 +
0.30 × 0.84 +
0.20 × 1.0 +
0.10 × 0.976
= 0.34 + 0.252 + 0.20 + 0.0976
= 0.890 ✅ (exceeds target)
V_meta (Prompt Quality)
Formula:
V_meta = 0.35 × completeness +
0.30 × clarity +
0.20 × adaptability +
0.15 × maintainability
Where:
- completeness = features_implemented / features_needed
- clarity = 1 - (ambiguous_instructions / total_instructions)
- adaptability = successful_task_types / tested_task_types
- maintainability = 1 - (prompt_complexity / max_complexity)
Target: V_meta ≥ 0.80
Example:
Completeness: 8/8 features = 1.0
Clarity: 1 - (2 ambiguous / 20 instructions) = 0.90
Adaptability: 5/6 task types = 0.83
Maintainability: 1 - (150 lines / 300 max) = 0.50
V_meta = 0.35 × 1.0 +
0.30 × 0.90 +
0.20 × 0.83 +
0.15 × 0.50
= 0.35 + 0.27 + 0.166 + 0.075
= 0.861 ✅ (exceeds target)
Metric Collection
Automated Collection
Session Analysis:
# Extract agent performance from session
query_tools --tool="Task" --scope=session | \
jq -r '.[] | select(.status == "success") | .duration' | \
awk '{sum+=$1; n++} END {print sum/n}'
Example Script:
#!/bin/bash
# scripts/measure-agent-metrics.sh
AGENT_NAME=$1
SESSION=$2
# Success rate
total=$(grep "agent=$AGENT_NAME" "$SESSION" | wc -l)
success=$(grep "agent=$AGENT_NAME.*success" "$SESSION" | wc -l)
success_rate=$(echo "scale=2; $success / $total" | bc)
# Average time
avg_time=$(grep "agent=$AGENT_NAME" "$SESSION" | \
jq -r '.duration' | \
awk '{sum+=$1; n++} END {print sum/n}')
# Quality (requires manual rating file)
avg_quality=$(cat "${SESSION}.ratings" | \
grep "$AGENT_NAME" | \
awk '{sum+=$2; n++} END {print sum/n}')
echo "Agent: $AGENT_NAME"
echo "Success Rate: $success_rate"
echo "Avg Time: ${avg_time}s"
echo "Avg Quality: $avg_quality/5"
Manual Collection
Test Suite Template:
## Agent Test Suite: [Agent Name]
**Iteration**: [N]
**Date**: [YYYY-MM-DD]
### Test Cases
| ID | Task | Result | Quality | Time | Notes |
|----|------|--------|---------|------|-------|
| 1 | [Description] | ✅/❌ | [1-5] | [min] | [Issues] |
| 2 | [Description] | ✅/❌ | [1-5] | [min] | [Issues] |
...
### Summary
- Success Rate: [X]% ([Y]/[Z])
- Avg Quality: [X.X]/5
- Avg Time: [X.X] min
- V_instance: [X.XX]
Benchmarking
Cross-Agent Comparison
Standard Test Suite: 20 representative tasks
Example Results:
| Agent | Success | Quality | Time | V_inst |
|-------------|---------|---------|-------|--------|
| Explore v1 | 60% | 3.1 | 4.2m | 0.62 |
| Explore v2 | 87.5% | 4.2 | 2.8m | 0.89 |
| Explore v3 | 90% | 4.3 | 2.6m | 0.91 |
Improvement: v1 → v3 = +30% success, +1.2 quality, +38% faster
Baseline Comparison
Industry Baselines (approximate):
- Generic agent (no tuning): ~50-60% success
- Basic tuned agent: ~70-80% success
- Well-tuned agent: ~85-95% success
- Expert-tuned agent: ~95-98% success
Regression Testing
Track Metrics Over Time
Regression Detection:
if current_metric < (previous_metric - threshold):
alert("REGRESSION DETECTED")
Thresholds:
- Success Rate: -5% (e.g., 90% → 85%)
- Quality Score: -0.3 (e.g., 4.5 → 4.2)
- Efficiency: +20% time (e.g., 2.8 min → 3.4 min)
Example:
Iteration 3: 90% success, 4.3 quality, 2.6 min ✅
Iteration 4: 87% success, 4.1 quality, 2.8 min ⚠️ REGRESSION
Analysis: New constraint too restrictive
Action: Revert constraint, re-test
Reporting Template
## Agent Metrics Report
**Agent**: [Name]
**Version**: [X.Y]
**Test Date**: [YYYY-MM-DD]
**Test Suite**: [Standard 20 | Custom N]
### Performance Metrics
**Success Rate**: [X]% ([Y]/[Z] tasks)
- Target: ≥85%
- Status: ✅/⚠️/❌
**Quality Score**: [X.X]/5
- Target: ≥4.0
- Status: ✅/⚠️/❌
**Efficiency**:
- Time: [X.X] min/task (target: [Y] min)
- Tokens: [X]k tokens/task (target: [Y]k)
- Status: ✅/⚠️/❌
**Reliability**: [X.XX]
- Target: ≥0.85
- Status: ✅/⚠️/❌
### Composite Scores
**V_instance**: [X.XX]
- Target: ≥0.80
- Status: ✅/⚠️/❌
**V_meta**: [X.XX]
- Target: ≥0.80
- Status: ✅/⚠️/❌
### Comparison to Baseline
| Metric | Baseline | Current | Δ |
|---------------|----------|---------|--------|
| Success Rate | [X]% | [Y]% | [+/-]% |
| Quality | [X.X] | [Y.Y] | [+/-] |
| Time | [X.X]m | [Y.Y]m | [+/-]% |
| V_instance | [X.XX] | [Y.YY] | [+/-] |
### Recommendations
1. [Action item based on metrics]
2. [Action item based on metrics]
### Next Steps
- [ ] [Task for next iteration]
- [ ] [Task for next iteration]
Source: BAIME Agent Prompt Evolution Framework Status: Production-ready, validated across 13 agent types Measurement Overhead: ~5 min per 20-task test suite