Initial commit
This commit is contained in:
Zhongwei Li
746
skills/orchestration-qa/examples.md
Normal file
746
skills/orchestration-qa/examples.md
Normal file
@@ -0,0 +1,746 @@
|
||||
# Orchestration QA Skill - Usage Examples
|
||||
|
||||
This document provides practical examples of using the orchestration-qa skill in different scenarios.
|
||||
|
||||
## Example 1: Interactive Configuration (FIRST TIME)
|
||||
|
||||
### Scenario
|
||||
First time using orchestration-qa in a session - choose which analyses to enable
|
||||
|
||||
### Usage
|
||||
```javascript
|
||||
orchestration-qa(phase="configure")
|
||||
```
|
||||
|
||||
### Interactive Prompts
|
||||
Agent presents multiselect options:
|
||||
|
||||
```markdown
|
||||
## Quality Analysis Configuration
|
||||
|
||||
**Which quality analysis categories would you like to enable for this session?**
|
||||
(Select all that apply)
|
||||
|
||||
☐ Information Density
|
||||
Analyze task content quality, detect wasteful patterns, measure information-to-token ratio (Specialists only)
|
||||
|
||||
☐ Execution Graphs
|
||||
Validate dependency graphs and parallel execution opportunities (Planning Specialist only)
|
||||
|
||||
☐ Tag Coverage
|
||||
Check tag consistency and agent-mapping coverage (Planning Specialist & Feature Architect)
|
||||
|
||||
☐ Token Optimization
|
||||
Identify token waste patterns (verbose output, unnecessary loading, redundant operations)
|
||||
|
||||
☐ Tool Selection
|
||||
Verify optimal tool usage (overview vs get, search vs filtered query, bulk operations)
|
||||
|
||||
☑ Routing Validation
|
||||
Detect Skills bypass violations (CRITICAL - status changes, feature creation, task execution)
|
||||
|
||||
☐ Parallel Detection
|
||||
Find missed parallelization opportunities (independent tasks, batch operations)
|
||||
|
||||
---
|
||||
|
||||
**How detailed should QA reports be?**
|
||||
|
||||
○ Brief - Only show critical issues (ALERT level) - minimal token usage
|
||||
● Standard - Show ALERT and WARN level issues with brief explanations
|
||||
○ Detailed - Show all issues (ALERT/WARN/INFO) with full analysis and recommendations
|
||||
```
|
||||
|
||||
### User Selections
|
||||
```javascript
|
||||
// User selects:
|
||||
{
|
||||
categories: ["Information Density", "Routing Validation"],
|
||||
reportStyle: "Standard"
|
||||
}
|
||||
```
|
||||
|
||||
### Output
|
||||
```markdown
|
||||
✅ **QA Configuration Saved**
|
||||
|
||||
**Enabled Categories (2):**
|
||||
- ✅ Information Density - Track task content quality and wasteful patterns
|
||||
- ✅ Routing Validation - Detect critical Skills bypass violations
|
||||
|
||||
**Disabled Categories (5):**
|
||||
- ⊘ Execution Graphs
|
||||
- ⊘ Tag Coverage
|
||||
- ⊘ Token Optimization
|
||||
- ⊘ Tool Selection
|
||||
- ⊘ Parallel Detection
|
||||
|
||||
**Report Style:** Standard (ALERT + WARN)
|
||||
|
||||
**Token Efficiency:**
|
||||
- Per analysis: ~1,200 tokens (vs ~3,500 comprehensive)
|
||||
- Savings: 67% reduction from full analysis
|
||||
- Trade-off: Focus on content quality + critical routing only
|
||||
|
||||
Configuration will be used for all QA reviews in this session.
|
||||
You can reconfigure anytime with `orchestration-qa(phase="configure")`
|
||||
```
|
||||
|
||||
## Example 2: Session Initialization (With Configuration)
|
||||
|
||||
### Scenario
|
||||
After configuration, initialize the session with knowledge bases
|
||||
|
||||
### Prerequisites
|
||||
Configuration must be set first (via `phase="configure"`)
|
||||
|
||||
### Usage
|
||||
```javascript
|
||||
orchestration-qa(phase="init")
|
||||
```
|
||||
|
||||
### Output
|
||||
```markdown
|
||||
✅ **Orchestration QA Initialized**
|
||||
|
||||
**Knowledge Base Loaded:**
|
||||
- Skills: 5 (feature-orchestration, task-orchestration, status-progression, dependency-analysis, dependency-orchestration)
|
||||
- Subagents: 8 (feature-architect, planning-specialist, backend-engineer, frontend-developer, database-engineer, test-engineer, technical-writer, bug-triage-specialist)
|
||||
- Routing: agent-mapping.yaml loaded (12 tag mappings)
|
||||
|
||||
**Active Configuration:**
|
||||
- ✅ Information Density (track task content quality)
|
||||
- ✅ Routing Validation (critical bypass detection)
|
||||
- ⊘ Execution Graphs (disabled)
|
||||
- ⊘ Tag Coverage (disabled)
|
||||
- ⊘ Token Optimization (disabled)
|
||||
- ⊘ Tool Selection (disabled)
|
||||
- ⊘ Parallel Detection (disabled)
|
||||
- Report Style: Standard (ALERT + WARN)
|
||||
|
||||
**Quality Assurance Active:**
|
||||
- ✅ Pre-execution validation
|
||||
- ✅ Post-execution review
|
||||
- ✅ Configuration-driven analysis (2 categories enabled)
|
||||
- ✅ Pattern tracking (continuous improvement)
|
||||
|
||||
**Session Tracking:**
|
||||
- Deviations: 0 ALERT, 0 WARN, 0 INFO
|
||||
- Patterns: 0 recurring issues
|
||||
- Improvements: 0 suggestions
|
||||
|
||||
Ready to monitor orchestration quality.
|
||||
**Token Efficiency:** ~1,200 tokens per analysis (67% savings from comprehensive)
|
||||
```
|
||||
|
||||
## Example 3: Feature Architect Validation (PRD Mode)
|
||||
|
||||
### Scenario
|
||||
User provides formal PRD document for feature creation
|
||||
|
||||
### User Request
|
||||
```markdown
|
||||
Create a user authentication feature:
|
||||
|
||||
# User Authentication
|
||||
|
||||
## Business Context
|
||||
We need secure user authentication for our platform.
|
||||
|
||||
## User Stories
|
||||
- As a user, I want to sign up with email/password
|
||||
- As a user, I want to log in securely
|
||||
- As a user, I want to reset my password
|
||||
|
||||
## Technical Requirements
|
||||
- OAuth2 support
|
||||
- JWT tokens
|
||||
- Role-based access control
|
||||
```
|
||||
|
||||
### Pre-Execution
|
||||
```javascript
|
||||
orchestration-qa(
|
||||
phase="pre",
|
||||
entityType="feature-architect",
|
||||
userInput="[full PRD above]"
|
||||
)
|
||||
```
|
||||
|
||||
### Pre-Execution Output
|
||||
```javascript
|
||||
{
|
||||
ready: true,
|
||||
contextCaptured: true,
|
||||
entityType: "feature-architect",
|
||||
category: "SUBAGENT",
|
||||
checkpoints: 8,
|
||||
expected: {
|
||||
mode: "PRD",
|
||||
prdSections: ["Business Context", "User Stories", "Technical Requirements"],
|
||||
descriptionLength: [500, 1000],
|
||||
sectionsExpected: 3,
|
||||
handoffTokens: [50, 100]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Post-Execution
|
||||
```javascript
|
||||
orchestration-qa(
|
||||
phase="post",
|
||||
entityType="feature-architect",
|
||||
entityOutput="[Feature Architect's response]",
|
||||
entityId="feature-uuid-123"
|
||||
)
|
||||
```
|
||||
|
||||
### Post-Execution Output (Success)
|
||||
```markdown
|
||||
## ✅ QA Review: Feature Architect (PRD Mode)
|
||||
|
||||
**Workflow Adherence:** 8/8 steps (100%)
|
||||
**Expected Outputs:** 7/7 present
|
||||
|
||||
### Validation Results
|
||||
- ✅ PRD mode detected correctly
|
||||
- ✅ All 3 PRD sections extracted to feature sections
|
||||
- ✅ Description forward-looking (623 chars)
|
||||
- ✅ Templates applied (Technical Approach, Requirements)
|
||||
- ✅ Tags follow project conventions (reused "authentication", "security")
|
||||
- ✅ agent-mapping.yaml checked (tags map to specialists)
|
||||
- ✅ Handoff minimal (87 tokens)
|
||||
- ✅ Core concepts preserved from PRD
|
||||
|
||||
**Quality Metrics:**
|
||||
- PRD extraction: 100% (3/3 sections)
|
||||
- Token efficiency: 95% (handoff 87 tokens, expected < 100)
|
||||
- Tag conventions: 100% (reused existing tags)
|
||||
|
||||
**Result:** Excellent execution - all patterns followed correctly!
|
||||
```
|
||||
|
||||
### Post-Execution Output (Issues Detected)
|
||||
```markdown
|
||||
## 🚨 QA Review: Feature Architect (PRD Mode) - ISSUES DETECTED
|
||||
|
||||
**Workflow Adherence:** 7/8 steps (87%)
|
||||
|
||||
### Critical Issues (1)
|
||||
|
||||
**❌ ALERT: PRD Sections Incomplete**
|
||||
|
||||
**What Happened:**
|
||||
Feature has 2 sections, but PRD contained 3 sections.
|
||||
|
||||
**Expected Behavior:**
|
||||
In PRD mode, Feature Architect must extract ALL sections from user's document.
|
||||
|
||||
**Impact:**
|
||||
"Technical Requirements" section from PRD was not transferred to feature.
|
||||
Requirements may be lost or incomplete.
|
||||
|
||||
**Evidence:**
|
||||
- PRD sections: ["Business Context", "User Stories", "Technical Requirements"]
|
||||
- Feature sections: ["Business Context", "User Stories"]
|
||||
- Missing: "Technical Requirements"
|
||||
|
||||
**Recommendation:**
|
||||
Add missing "Technical Requirements" section to feature.
|
||||
|
||||
**Definition Update Needed:**
|
||||
Update feature-architect.md Step 7 to include validation:
|
||||
- [ ] Verify all PRD sections extracted
|
||||
- [ ] Compare PRD section count vs feature section count
|
||||
- [ ] If mismatch, add missing sections before returning
|
||||
|
||||
---
|
||||
|
||||
### ✅ Successes (6)
|
||||
- PRD mode detected correctly
|
||||
- Description forward-looking
|
||||
- Templates applied
|
||||
- Tags follow conventions
|
||||
- agent-mapping.yaml checked
|
||||
- Handoff minimal (92 tokens)
|
||||
|
||||
### 📋 Added to TodoWrite
|
||||
- [ ] Add "Technical Requirements" section to Feature [ID]
|
||||
- [ ] Update feature-architect.md Step 7 validation checklist
|
||||
|
||||
### 💭 Decision Required
|
||||
|
||||
**Question:** Should we add the missing "Technical Requirements" section now?
|
||||
|
||||
**Options:**
|
||||
1. **Add section now** ✅ Recommended
|
||||
- Pros: Ensures all PRD content captured
|
||||
- Cons: Requires one additional tool call
|
||||
|
||||
2. **Accept as-is**
|
||||
- Pros: Faster (no additional work)
|
||||
- Cons: Requirements may be incomplete
|
||||
|
||||
**Recommendation:** Add section now to ensure complete PRD capture.
|
||||
|
||||
**Your choice?**
|
||||
```
|
||||
|
||||
## Example 4: Planning Specialist Review (Graph Analysis)
|
||||
|
||||
### Scenario
|
||||
Planning Specialist breaks down feature into tasks
|
||||
|
||||
### Post-Execution
|
||||
```javascript
|
||||
orchestration-qa(
|
||||
phase="post",
|
||||
entityType="planning-specialist",
|
||||
entityOutput="[Planning Specialist's response]",
|
||||
entityId="feature-uuid-123"
|
||||
)
|
||||
```
|
||||
|
||||
### Output (High Quality)
|
||||
```markdown
|
||||
## ✅ QA Review: Planning Specialist
|
||||
|
||||
**Workflow Adherence:** 8/8 steps (100%)
|
||||
**Expected Outputs:** 7/7 present
|
||||
|
||||
### Specialized Analysis
|
||||
|
||||
**📊 Execution Graph Quality: 98%**
|
||||
- Dependency Accuracy: 100% (all dependencies correct)
|
||||
- Parallel Completeness: 100% (all opportunities identified)
|
||||
- Format Clarity: 95% (clear batch numbers, explicit dependencies)
|
||||
|
||||
**🏷️ Tag Quality: 100%**
|
||||
- Tag Coverage: 100% (all tasks have tags)
|
||||
- Agent Mapping Coverage: 100% (all tags map to specialists)
|
||||
- Convention Adherence: 100% (reused existing tags)
|
||||
|
||||
### Quality Metrics
|
||||
- Domain isolation: ✅ (one task = one specialist)
|
||||
- Dependencies mapped: ✅ (Database → Backend → Frontend pattern)
|
||||
- Documentation task: ✅ (user-facing feature)
|
||||
- Testing task: ✅ (created)
|
||||
- No circular dependencies: ✅
|
||||
- Templates applied: ✅
|
||||
|
||||
**Result:** Excellent execution - target quality (95%+) achieved!
|
||||
```
|
||||
|
||||
### Output (Issues Detected)
|
||||
```markdown
|
||||
## ⚠️ QA Review: Planning Specialist - Issues Found
|
||||
|
||||
**Workflow Adherence:** 8/8 steps (100%)
|
||||
|
||||
### Specialized Analysis
|
||||
|
||||
**📊 Execution Graph Quality: 73%**
|
||||
- Dependency Accuracy: 67% (2/3 dependencies incorrect)
|
||||
- Parallel Completeness: 67% (1 opportunity missed)
|
||||
- Format Clarity: 85% (some ambiguous notation)
|
||||
|
||||
**🏷️ Tag Quality: 92%**
|
||||
- Tag Coverage: 100% (all tasks have tags)
|
||||
- Agent Mapping Coverage: 100% (all tags map to specialists)
|
||||
- Convention Adherence: 75% (1 new tag without agent-mapping check)
|
||||
|
||||
### Issues Detected (4)
|
||||
|
||||
**🚨 ALERT: Incorrect Dependency**
|
||||
- Task: "Implement backend API"
|
||||
- Expected blocked by: ["Create database schema", "Design API endpoints"]
|
||||
- Found in graph: ["Design API endpoints"]
|
||||
- **Missing:** "Create database schema"
|
||||
- **Impact:** Task might start before database ready
|
||||
|
||||
**🚨 ALERT: Cross-Domain Task**
|
||||
- Task: "Build authentication UI"
|
||||
- Domains detected: frontend + backend
|
||||
- Evidence: Description mentions "UI components AND API integration"
|
||||
- **Fix:** Split into "Build authentication UI" (frontend) + "Integrate auth API" (backend)
|
||||
|
||||
**⚠️ WARN: Missed Parallel Opportunity**
|
||||
- Task: "Create database schema"
|
||||
- Status: Independent (no blockers)
|
||||
- **Issue:** Not in Batch 1, placed in Batch 2
|
||||
- **Fix:** Move to Batch 1 (can start immediately)
|
||||
|
||||
**⚠️ WARN: Tag Convention Violation**
|
||||
- Task: "Build authentication UI"
|
||||
- Tag: "auth-ui" (new tag)
|
||||
- **Issue:** Tag not in project conventions, not checked in agent-mapping.yaml
|
||||
- **Suggestion:** Use existing tag "frontend" or add "auth-ui" to agent-mapping.yaml
|
||||
|
||||
### ✅ Successes
|
||||
- 8/8 workflow steps completed
|
||||
- Documentation task included
|
||||
- Testing task included
|
||||
- Task descriptions populated (200-600 chars)
|
||||
|
||||
### 📋 Added to TodoWrite
|
||||
- [ ] Fix dependency: "Implement backend API" should depend on "Create database schema"
|
||||
- [ ] Split cross-domain task: "Build authentication UI"
|
||||
- [ ] Move "Create database schema" to Batch 1
|
||||
- [ ] Review tag "auth-ui" - add to agent-mapping.yaml or use "frontend"
|
||||
|
||||
### 🎯 Recommendations
|
||||
1. Update dependencies (CRITICAL - affects execution order)
|
||||
2. Split cross-domain task (CRITICAL - routing will fail)
|
||||
3. Optimize parallel execution (move task to Batch 1)
|
||||
4. Update planning-specialist.md Step 3 to enforce domain isolation check
|
||||
```
|
||||
|
||||
## Example 5: Status Progression Skill Validation
|
||||
|
||||
### Scenario
|
||||
Orchestrator attempts to mark task complete
|
||||
|
||||
### Pre-Execution (Routing Violation Detected)
|
||||
```javascript
|
||||
orchestration-qa(
|
||||
phase="pre",
|
||||
entityType="manage_container", // Direct tool call detected
|
||||
userInput="Mark task T1 complete"
|
||||
)
|
||||
```
|
||||
|
||||
### Output (Immediate Alert)
|
||||
```markdown
|
||||
## 🚨 CRITICAL ROUTING VIOLATION
|
||||
|
||||
**Severity:** CRITICAL
|
||||
|
||||
**Workflow Type:** status-change
|
||||
|
||||
**Violation:** Status change bypassed mandatory Status Progression Skill
|
||||
|
||||
**Expected Approach:** Use Status Progression Skill
|
||||
|
||||
**Actual Approach:** Direct manage_container(operation="setStatus") call
|
||||
|
||||
**Impact:** Prerequisite validation may be skipped:
|
||||
- Summary length validation (300-500 chars required)
|
||||
- Dependency completion check
|
||||
- Files Changed section verification (for code tasks)
|
||||
- Test validation
|
||||
|
||||
**Recommendation:** Use Status Progression Skill for ALL status changes
|
||||
|
||||
---
|
||||
|
||||
**Action:** STOP - Do not proceed with direct status change
|
||||
|
||||
**Correct Approach:**
|
||||
```javascript
|
||||
Use status-progression skill with task ID
|
||||
```
|
||||
|
||||
**Why This Matters:**
|
||||
Status Progression Skill validates prerequisites before allowing status change.
|
||||
Bypassing this validation can result in incomplete tasks being marked complete.
|
||||
```
|
||||
|
||||
## Example 6: Implementation Specialist Review
|
||||
|
||||
### Scenario
|
||||
Backend Engineer completes task
|
||||
|
||||
### Post-Execution
|
||||
```javascript
|
||||
orchestration-qa(
|
||||
phase="post",
|
||||
entityType="backend-engineer",
|
||||
entityOutput="[Backend Engineer's response]",
|
||||
entityId="task-uuid-456"
|
||||
)
|
||||
```
|
||||
|
||||
### Output (Success)
|
||||
```markdown
|
||||
## ✅ QA Review: Backend Engineer
|
||||
|
||||
**Workflow Adherence:** 9/9 steps (100%)
|
||||
|
||||
### Lifecycle Validation
|
||||
- ✅ Read task with sections
|
||||
- ✅ Read dependencies
|
||||
- ✅ Completed implementation work
|
||||
- ✅ Updated task sections with results
|
||||
- ✅ Tests run and passing
|
||||
- ✅ Summary populated (387 chars)
|
||||
- ✅ Files Changed section created (ordinal 999)
|
||||
- ✅ Used Status Progression Skill to mark complete
|
||||
- ✅ Output minimal (73 tokens)
|
||||
|
||||
### Quality Checks
|
||||
- Summary length: 387 chars (expected 300-500) ✅
|
||||
- Files Changed: Present ✅
|
||||
- Tests mentioned: Yes ("All 12 tests passing") ✅
|
||||
- Status change method: Status Progression Skill ✅
|
||||
- Output brevity: 73 tokens (expected 50-100) ✅
|
||||
|
||||
**Result:** Perfect lifecycle execution!
|
||||
```
|
||||
|
||||
### Output (Issues)
|
||||
```markdown
|
||||
## 🚨 QA Review: Backend Engineer - CRITICAL ISSUE
|
||||
|
||||
**Workflow Adherence:** 8/9 steps (89%)
|
||||
|
||||
### Critical Issues (1)
|
||||
|
||||
**❌ ALERT: Marked Complete Without Status Progression Skill**
|
||||
|
||||
**What Happened:**
|
||||
Backend Engineer called manage_container(operation="setStatus") directly.
|
||||
|
||||
**Expected Behavior:**
|
||||
Step 8 of specialist lifecycle requires using Status Progression Skill.
|
||||
|
||||
**Impact:**
|
||||
- Summary validation may have been skipped (no length check)
|
||||
- Files Changed section may not have been verified
|
||||
- Test validation may have been incomplete
|
||||
|
||||
**Evidence:**
|
||||
- Task status changed to "completed"
|
||||
- No mention of "Status Progression" in output
|
||||
- Direct tool call detected
|
||||
|
||||
**Recommendation:**
|
||||
All implementation specialists MUST use Status Progression Skill in Step 8.
|
||||
|
||||
**Definition Update Needed:**
|
||||
Update backend-engineer.md to emphasize CRITICAL pattern:
|
||||
```diff
|
||||
### Step 8: Use Status Progression Skill to Mark Complete
|
||||
|
||||
+ **CRITICAL:** NEVER call manage_container directly for status changes
|
||||
+ **ALWAYS:** Use Status Progression Skill for prerequisite validation
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ⚠️ Issues (1)
|
||||
|
||||
**⚠️ WARN: Files Changed Section Missing**
|
||||
- Expected: Section with ordinal 999, title "Files Changed"
|
||||
- Found: No Files Changed section
|
||||
- **Impact:** Difficult to track what files were modified
|
||||
|
||||
### 📋 Added to TodoWrite
|
||||
- [ ] ALERT: Backend Engineer bypassed Status Progression Skill
|
||||
- [ ] Add Files Changed section to task
|
||||
- [ ] Update backend-engineer.md Step 8 critical pattern
|
||||
|
||||
### 💭 Decision Required
|
||||
|
||||
**Issue:** Critical workflow pattern violated (Status Progression bypass)
|
||||
|
||||
**Options:**
|
||||
1. **Validate task manually**
|
||||
- Check summary length (300-500 chars)
|
||||
- Verify Files Changed section exists or create it
|
||||
- Confirm tests passing
|
||||
|
||||
2. **Revert and retry with Status Progression Skill**
|
||||
- Revert task to "in-progress"
|
||||
- Use Status Progression Skill for completion
|
||||
- Ensures all prerequisites validated
|
||||
|
||||
**Recommendation:** Option 1 for this instance, but update backend-engineer.md
|
||||
to prevent recurrence.
|
||||
```
|
||||
|
||||
## Example 7: Session Summary with Patterns
|
||||
|
||||
### Scenario
|
||||
End of session after multiple workflows
|
||||
|
||||
### Usage
|
||||
```javascript
|
||||
orchestration-qa(phase="summary", sessionId="session-123")
|
||||
```
|
||||
|
||||
### Output
|
||||
```markdown
|
||||
## 📊 Session QA Summary
|
||||
|
||||
**Workflows Analyzed:** 6
|
||||
- Skills: 2 (Feature Orchestration, Status Progression)
|
||||
- Subagents: 4 (Feature Architect, Planning Specialist, 2x Backend Engineer)
|
||||
|
||||
**Quality Overview:**
|
||||
- ✅ Successful: 4 (no issues)
|
||||
- ⚠️ Issues: 1 (Planning Specialist - graph quality 73%)
|
||||
- 🚨 Critical: 1 (Backend Engineer - status bypass)
|
||||
|
||||
### Deviation Breakdown
|
||||
- Routing violations: 1 (status change bypass)
|
||||
- Workflow deviations: 0
|
||||
- Output quality: 0
|
||||
- Dependency errors: 2 (in Planning Specialist)
|
||||
- Tag issues: 1 (convention violation)
|
||||
- Token waste: 0
|
||||
|
||||
### Recurring Patterns (1)
|
||||
|
||||
**🔁 Pattern: Status Change Bypasses**
|
||||
- Occurrences: 2 (Backend Engineer x2)
|
||||
- Root cause: Step 8 critical pattern not emphasized enough
|
||||
- Impact: Prerequisites validation skipped
|
||||
- **Suggestion**: Update backend-engineer.md Step 8 with CRITICAL emphasis
|
||||
|
||||
### Improvement Recommendations (2)
|
||||
|
||||
**Priority 1: Backend Engineer Definition Update**
|
||||
- File: backend-engineer.md
|
||||
- Section: Step 8
|
||||
- Type: Critical Pattern Emphasis
|
||||
- Change: Add CRITICAL warning against direct status changes
|
||||
- Impact: Prevents status bypass in future executions
|
||||
- Effort: Low (text addition)
|
||||
|
||||
**Priority 2: Planning Specialist Validation Checklist**
|
||||
- File: planning-specialist.md
|
||||
- Section: Step 5 (Map Dependencies)
|
||||
- Type: Validation Checklist
|
||||
- Change: Add graph quality validation before returning
|
||||
- Impact: Ensures execution graph accuracy >= 95%
|
||||
- Effort: Medium (add quality gate step)
|
||||
|
||||
### Quality Trends
|
||||
- Graph quality: 87% average (baseline 70%, target 95%+)
|
||||
- Tag coverage: 98% average (baseline 90%, target 100%)
|
||||
- Token efficiency: 91% average
|
||||
- Workflow adherence: 94% average
|
||||
|
||||
### Next Steps
|
||||
1. Update backend-engineer.md Step 8 (CRITICAL pattern emphasis)
|
||||
2. Update planning-specialist.md Step 5 (graph validation checklist)
|
||||
3. Monitor for recurrence in next session
|
||||
```
|
||||
|
||||
## Example 8: Mid-Session Reconfiguration
|
||||
|
||||
### Scenario
|
||||
User wants to enable additional analysis categories after session has started
|
||||
|
||||
### User Request
|
||||
"I want to also track token optimization now to see if we're wasting tokens"
|
||||
|
||||
### Usage
|
||||
```javascript
|
||||
orchestration-qa(phase="configure")
|
||||
```
|
||||
|
||||
### Interactive Prompts
|
||||
Agent presents options with current configuration pre-selected:
|
||||
|
||||
```markdown
|
||||
## Quality Analysis Configuration
|
||||
|
||||
**Current Configuration:**
|
||||
- ✅ Information Density
|
||||
- ✅ Routing Validation
|
||||
|
||||
**Which quality analysis categories would you like to enable?**
|
||||
(Your current selections are pre-checked)
|
||||
|
||||
☑ Information Density ← Currently enabled
|
||||
☐ Execution Graphs
|
||||
☐ Tag Coverage
|
||||
☑ Token Optimization ← User adds this
|
||||
☐ Tool Selection
|
||||
☑ Routing Validation ← Currently enabled
|
||||
☐ Parallel Detection
|
||||
```
|
||||
|
||||
### User Updates
|
||||
```javascript
|
||||
// User selects:
|
||||
{
|
||||
categories: ["Information Density", "Routing Validation", "Token Optimization"],
|
||||
reportStyle: "Standard" // Keep existing
|
||||
}
|
||||
```
|
||||
|
||||
### Output
|
||||
```markdown
|
||||
✅ **QA Configuration Updated**
|
||||
|
||||
**Changes:**
|
||||
- ➕ Added: Token Optimization
|
||||
|
||||
**New Configuration (3 categories):**
|
||||
- ✅ Information Density
|
||||
- ✅ Routing Validation
|
||||
- ✅ Token Optimization (NEW)
|
||||
|
||||
**Report Style:** Standard (unchanged)
|
||||
|
||||
**Token Impact:**
|
||||
- Previous: ~1,200 tokens per analysis
|
||||
- Updated: ~1,800 tokens per analysis (+600 tokens)
|
||||
- Trade-off: Now detecting token waste patterns
|
||||
|
||||
All future QA reviews will include token optimization analysis.
|
||||
```
|
||||
|
||||
### Next Post-Execution
|
||||
```javascript
|
||||
// After Backend Engineer completes task
|
||||
orchestration-qa(
|
||||
phase="post",
|
||||
entityType="backend-engineer",
|
||||
entityOutput="...",
|
||||
entityId="task-uuid"
|
||||
)
|
||||
|
||||
// Now loads: post-execution.md + routing-validation.md +
|
||||
// task-content-quality.md + token-optimization.md
|
||||
// Analysis includes: content quality + routing + token waste detection
|
||||
```
|
||||
|
||||
## Usage Patterns Summary
|
||||
|
||||
### Session Start (First Time)
|
||||
1. `phase="configure"` - Interactive category selection (~200-300 tokens)
|
||||
2. `phase="init"` - Load knowledge bases (~1000 tokens)
|
||||
|
||||
### Per Entity
|
||||
- `phase="pre"` - Before launching any Skill or Subagent (~600 tokens)
|
||||
- `phase="post"` - After any Skill or Subagent completes (varies by config)
|
||||
|
||||
### Optional
|
||||
- `phase="configure"` - Reconfigure mid-session
|
||||
- `phase="summary"` - End-of-session pattern tracking (~800 tokens)
|
||||
|
||||
### Configuration-Driven Token Costs
|
||||
|
||||
**Post-Execution Costs by Configuration:**
|
||||
|
||||
| Configuration | Token Cost | Use Case |
|
||||
|--------------|------------|----------|
|
||||
| **Minimal** (Routing only) | ~1,000 tokens | Critical bypass detection only |
|
||||
| **Default** (Info Density + Routing) | ~1,200 tokens | Most users - content + critical checks |
|
||||
| **Planning Focus** (Graphs + Tags + Routing) | ~2,000 tokens | Planning Specialist reviews |
|
||||
| **Comprehensive** (All enabled) | ~3,500 tokens | Full quality analysis |
|
||||
|
||||
**Session Cost Examples:**
|
||||
|
||||
| Workflow | Config | Total Cost | vs Monolithic |
|
||||
|----------|--------|------------|---------------|
|
||||
| 1 Feature + 3 Tasks | Default | ~6k tokens | 70% savings |
|
||||
| 1 Feature + 3 Tasks | Minimal | ~4.5k tokens | 78% savings |
|
||||
| 1 Feature + 3 Tasks | Comprehensive | ~15k tokens | 25% savings |
|
||||
|
||||
**Monolithic Trainer**: 20k-30k tokens always loaded (no configuration)
|
||||
|
||||
**Smart Defaults**: Information Density + Routing Validation achieves 93% token reduction while catching critical issues
|
||||
Reference in New Issue
Block a user