688 lines
22 KiB
Markdown
688 lines
22 KiB
Markdown
# ADW Architecture Deep Dive
|
|
|
|
## System Overview
|
|
|
|
ADW infrastructure creates a **programmatic interface** for AI-driven development by wrapping the application layer with an agentic layer that templates engineering patterns.
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Agentic Layer │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ ADW Scripts (adws/) │ │
|
|
│ │ - Execute prompts programmatically │ │
|
|
│ │ - Orchestrate multi-phase workflows │ │
|
|
│ │ - Invoke Claude Code CLI or SDK │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ Slash Commands (.claude/commands/) │ │
|
|
│ │ - Reusable prompt templates │ │
|
|
│ │ - Structured instructions for agents │ │
|
|
│ │ - Variable substitution │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ Specifications (specs/) │ │
|
|
│ │ - Implementation plans │ │
|
|
│ │ - Step-by-step tasks │ │
|
|
│ │ - Validation commands │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ Claude Code (subprocess or SDK) │ │
|
|
│ │ - Executes prompts with tools │ │
|
|
│ │ - Modifies application code │ │
|
|
│ │ - Returns structured results │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
│ ↓ │
|
|
│ ┌───────────────────────────────────────────────────────┐ │
|
|
│ │ Observability (agents/) │ │
|
|
│ │ - Structured outputs │ │
|
|
│ │ - Execution tracking │ │
|
|
│ │ - Debug artifacts │ │
|
|
│ └───────────────────────────────────────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────────────────────────────┐
|
|
│ Application Layer │
|
|
│ - Your actual application code (apps/, src/, lib/) │
|
|
│ - Tests, configs, documentation │
|
|
│ - What agents read and modify │
|
|
└─────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
## Core Modules
|
|
|
|
### agent.py - Subprocess Execution Engine
|
|
|
|
**Purpose**: Core module for executing Claude Code CLI as subprocess
|
|
|
|
**Key Components**:
|
|
|
|
```python
|
|
# Data Models
|
|
class AgentPromptRequest(BaseModel):
|
|
prompt: str # What to execute
|
|
adw_id: str # Unique workflow ID
|
|
agent_name: str # Agent identifier
|
|
model: Literal["sonnet", "opus"]
|
|
output_file: str # Where to save output
|
|
working_dir: Optional[str] # Where to run
|
|
|
|
class AgentPromptResponse(BaseModel):
|
|
output: str # Result text
|
|
success: bool # Did it succeed?
|
|
session_id: Optional[str] # Claude session ID
|
|
retry_code: RetryCode # Should we retry?
|
|
|
|
# Core Functions
|
|
def prompt_claude_code(request: AgentPromptRequest) -> AgentPromptResponse
|
|
def prompt_claude_code_with_retry(request: AgentPromptRequest) -> AgentPromptResponse
|
|
def execute_template(request: AgentTemplateRequest) -> AgentPromptResponse
|
|
```
|
|
|
|
**Execution Flow**:
|
|
|
|
1. **Build Command**
|
|
```python
|
|
cmd = [
|
|
"claude",
|
|
"-p", prompt,
|
|
"--model", model,
|
|
"--output-format", "stream-json",
|
|
"--verbose"
|
|
]
|
|
```
|
|
|
|
2. **Filter Environment**
|
|
```python
|
|
env = get_safe_subprocess_env()
|
|
# Only passes essential variables
|
|
# Prevents environment leakage
|
|
```
|
|
|
|
3. **Execute & Stream**
|
|
```python
|
|
subprocess.run(
|
|
cmd,
|
|
stdout=output_file, # Stream to file
|
|
stderr=subprocess.PIPE,
|
|
env=env
|
|
)
|
|
```
|
|
|
|
4. **Parse JSONL Output**
|
|
```python
|
|
messages = [json.loads(line) for line in f]
|
|
result_message = find_result_message(messages)
|
|
```
|
|
|
|
5. **Convert to Multiple Formats**
|
|
- `cc_raw_output.jsonl` - Raw streaming JSONL
|
|
- `cc_raw_output.json` - Parsed JSON array
|
|
- `cc_final_object.json` - Final result object
|
|
|
|
6. **Return Response**
|
|
```python
|
|
return AgentPromptResponse(
|
|
output=result_text,
|
|
success=not is_error,
|
|
session_id=session_id,
|
|
retry_code=RetryCode.NONE
|
|
)
|
|
```
|
|
|
|
**Error Handling**:
|
|
- Retry codes distinguish transient from permanent errors
|
|
- Output truncation prevents console flooding
|
|
- Graceful degradation on parse failures
|
|
|
|
### agent_sdk.py - SDK Execution Engine
|
|
|
|
**Purpose**: Type-safe, async execution using Claude Code Python SDK
|
|
|
|
**Key Patterns**:
|
|
|
|
```python
|
|
# One-shot query
|
|
async def simple_query(prompt: str) -> str:
|
|
options = ClaudeCodeOptions(model="claude-sonnet-4-20250514")
|
|
texts = []
|
|
async for message in query(prompt=prompt, options=options):
|
|
if isinstance(message, AssistantMessage):
|
|
texts.append(extract_text(message))
|
|
return "\n".join(texts)
|
|
|
|
# Interactive session
|
|
@asynccontextmanager
|
|
async def create_session():
|
|
client = ClaudeSDKClient(options=options)
|
|
await client.connect()
|
|
try:
|
|
yield client
|
|
finally:
|
|
await client.disconnect()
|
|
```
|
|
|
|
**Advantages**:
|
|
- Native async/await patterns
|
|
- Typed message objects (AssistantMessage, ResultMessage)
|
|
- SDK-specific error handling
|
|
- Interactive session support
|
|
- Better IDE integration
|
|
|
|
**When to Use**:
|
|
- Interactive multi-turn conversations
|
|
- Need better type safety
|
|
- Async workflow integration
|
|
- Native Python integration preferred
|
|
|
|
## Data Flow Patterns
|
|
|
|
### Pattern 1: Direct Prompt Execution
|
|
|
|
```
|
|
User Input
|
|
↓
|
|
./adws/adw_prompt.py "analyze this"
|
|
↓
|
|
AgentPromptRequest created
|
|
↓
|
|
prompt_claude_code_with_retry()
|
|
↓
|
|
subprocess: claude -p "analyze this"
|
|
↓
|
|
JSONL output streamed to file
|
|
↓
|
|
Parse JSONL → JSON conversion
|
|
↓
|
|
AgentPromptResponse returned
|
|
↓
|
|
Display to user + save summary
|
|
```
|
|
|
|
### Pattern 2: Slash Command Execution
|
|
|
|
```
|
|
User Input
|
|
↓
|
|
./adws/adw_slash_command.py /chore abc123 "task"
|
|
↓
|
|
Read .claude/commands/chore.md
|
|
↓
|
|
Substitute variables ($1, $2)
|
|
↓
|
|
AgentTemplateRequest created
|
|
↓
|
|
execute_template() → prompt_claude_code_with_retry()
|
|
↓
|
|
subprocess: claude -p "<expanded template>"
|
|
↓
|
|
Parse results
|
|
↓
|
|
Extract spec path from output
|
|
↓
|
|
Display + save
|
|
```
|
|
|
|
### Pattern 3: Compound Workflow
|
|
|
|
```
|
|
User Input
|
|
↓
|
|
./adws/adw_chore_implement.py "feature X"
|
|
↓
|
|
Generate unique adw_id
|
|
↓
|
|
┌─────────────────────────────────────┐
|
|
│ Phase 1: Planning │
|
|
├─────────────────────────────────────┤
|
|
│ execute_template("/chore", args) │
|
|
│ ↓ │
|
|
│ Create plan in specs/ │
|
|
│ ↓ │
|
|
│ Parse output for plan path │
|
|
└─────────────────────────────────────┘
|
|
↓
|
|
┌─────────────────────────────────────┐
|
|
│ Phase 2: Implementation │
|
|
├─────────────────────────────────────┤
|
|
│ execute_template("/implement", path) │
|
|
│ ↓ │
|
|
│ Read plan from specs/ │
|
|
│ ↓ │
|
|
│ Execute step by step │
|
|
│ ↓ │
|
|
│ Modify application code │
|
|
└─────────────────────────────────────┘
|
|
↓
|
|
Workflow summary saved
|
|
↓
|
|
Display results to user
|
|
```
|
|
|
|
## Directory Structure
|
|
|
|
### Minimal Setup
|
|
|
|
```
|
|
project/
|
|
├── adws/
|
|
│ ├── adw_modules/
|
|
│ │ └── agent.py # Core execution
|
|
│ └── adw_prompt.py # CLI wrapper
|
|
├── .claude/commands/
|
|
│ ├── chore.md # Planning template
|
|
│ └── implement.md # Implementation template
|
|
├── specs/
|
|
│ └── *.md # Generated plans
|
|
├── agents/ # Output directory
|
|
│ └── {adw_id}/
|
|
│ └── {agent_name}/
|
|
│ ├── cc_raw_output.jsonl
|
|
│ ├── cc_raw_output.json
|
|
│ ├── cc_final_object.json
|
|
│ └── custom_summary_output.json
|
|
└── .env.sample # Configuration template
|
|
```
|
|
|
|
### Enhanced Setup
|
|
|
|
```
|
|
project/
|
|
├── adws/
|
|
│ ├── adw_modules/
|
|
│ │ ├── agent.py # Subprocess execution
|
|
│ │ └── agent_sdk.py # SDK execution
|
|
│ ├── adw_prompt.py # Direct prompts
|
|
│ ├── adw_slash_command.py # Command executor
|
|
│ └── adw_chore_implement.py # Compound workflow
|
|
├── .claude/commands/
|
|
│ ├── chore.md
|
|
│ ├── implement.md
|
|
│ ├── feature.md # Feature planning
|
|
│ ├── test.md # Test creation
|
|
│ ├── prime.md # Context loading
|
|
│ └── start.md # App startup
|
|
├── specs/
|
|
│ ├── chore-*.md
|
|
│ └── feature-*.md
|
|
├── agents/
|
|
│ └── {adw_id}/
|
|
│ ├── planner/ # Planning agent outputs
|
|
│ ├── builder/ # Building agent outputs
|
|
│ └── workflow_summary.json # Overall summary
|
|
└── CLAUDE.md # Updated with ADW docs
|
|
```
|
|
|
|
### Scaled Setup (Production)
|
|
|
|
```
|
|
project/
|
|
├── adws/
|
|
│ ├── adw_modules/
|
|
│ │ ├── agent.py # Subprocess execution
|
|
│ │ ├── agent_sdk.py # SDK execution
|
|
│ │ ├── data_types.py # Type definitions
|
|
│ │ ├── state.py # State management (adw_state.json)
|
|
│ │ ├── git_ops.py # Git operations (branch, commit, push, PR)
|
|
│ │ ├── worktree_ops.py # Worktree isolation management
|
|
│ │ ├── workflow_ops.py # High-level orchestration
|
|
│ │ ├── github.py # GitHub integration (gh CLI)
|
|
│ │ └── utils.py # Shared utilities
|
|
│ ├── adw_sdlc_iso.py # Complete SDLC (plan→build→test→review→doc)
|
|
│ ├── adw_plan_build_test_review_iso.py # Compound isolated workflow
|
|
│ ├── adw_ship_iso.py # Merge validation and shipping
|
|
│ ├── adw_plan_iso.py # Isolated planning phase
|
|
│ ├── adw_build_iso.py # Isolated build phase
|
|
│ ├── adw_test_iso.py # Isolated testing phase
|
|
│ ├── adw_review_iso.py # Isolated review phase
|
|
│ └── adw_document_iso.py # Isolated documentation phase
|
|
├── .claude/commands/
|
|
│ ├── chore.md
|
|
│ ├── bug.md
|
|
│ ├── feature.md
|
|
│ ├── implement.md
|
|
│ ├── classify_issue.md # Issue classification
|
|
│ ├── classify_adw.md # ADW workflow selection
|
|
│ ├── generate_branch_name.md # Branch naming
|
|
│ ├── patch.md # Patch planning
|
|
│ ├── test.md # Test execution
|
|
│ ├── review.md # Code review
|
|
│ ├── document.md # Documentation generation
|
|
│ ├── pull_request.md # PR creation
|
|
│ ├── install_worktree.md # Worktree environment setup
|
|
│ ├── cleanup_worktrees.md # Worktree cleanup
|
|
│ ├── commit.md # Commit message generation
|
|
│ ├── prime.md
|
|
│ └── start.md
|
|
├── specs/
|
|
│ ├── chore-*.md
|
|
│ ├── bug-*.md
|
|
│ ├── feature-*.md
|
|
│ └── patch/ # Patch plans
|
|
├── agents/
|
|
│ └── {adw_id}/
|
|
│ ├── {agent_name}/ # Per-agent outputs
|
|
│ ├── adw_state.json # Persistent state
|
|
│ └── workflow_summary.json
|
|
├── trees/ # Git worktree isolation
|
|
│ └── {adw_id}/ # Isolated working directory
|
|
│ ├── .ports.env # Port configuration
|
|
│ └── <full repo copy>
|
|
├── .adw_backups/ # Safety backups during upgrades
|
|
├── CLAUDE.md # Updated with ADW docs
|
|
└── .env.sample # Configuration
|
|
```
|
|
|
|
**Key Scaled Phase Features:**
|
|
|
|
1. **State Management**: Persistent state across workflow phases via `adw_state.json`
|
|
2. **Git Worktree Isolation**: Each ADW runs in isolated `trees/{adw_id}/` directory
|
|
3. **Port Management**: Deterministic port allocation (9100-9114 backend, 9200-9214 frontend)
|
|
4. **GitHub Integration**: Issue operations, PR management, comment posting via gh CLI
|
|
5. **Multi-Phase Workflows**: Complete SDLC automation (plan, build, test, review, document, ship)
|
|
6. **Workflow Composition**: High-level functions for issue classification, branch generation, etc.
|
|
7. **Advanced Commands**: Rich library of 20+ specialized slash commands
|
|
|
|
## Output Structure
|
|
|
|
### Observability Artifacts
|
|
|
|
Each ADW execution creates:
|
|
|
|
**cc_raw_output.jsonl**
|
|
- Line-delimited JSON
|
|
- Streaming output from Claude Code
|
|
- Each line is a message object
|
|
- Last line is result message
|
|
|
|
**cc_raw_output.json**
|
|
- JSON array of all messages
|
|
- Easier for programmatic processing
|
|
- Contains full conversation history
|
|
|
|
**cc_final_object.json**
|
|
- Just the last message (result)
|
|
- Quick access to final output
|
|
- Contains success/failure info
|
|
|
|
**custom_summary_output.json**
|
|
- High-level execution summary
|
|
- Metadata (adw_id, prompt, model)
|
|
- Success status
|
|
- Session ID for debugging
|
|
|
|
### Workflow Tracking
|
|
|
|
For compound workflows:
|
|
|
|
**workflow_summary.json**
|
|
```json
|
|
{
|
|
"workflow": "chore_implement",
|
|
"adw_id": "abc12345",
|
|
"prompt": "add error handling",
|
|
"phases": {
|
|
"planning": {
|
|
"success": true,
|
|
"session_id": "session_xyz",
|
|
"agent": "planner",
|
|
"output_dir": "agents/abc12345/planner/"
|
|
},
|
|
"implementation": {
|
|
"success": true,
|
|
"session_id": "session_abc",
|
|
"agent": "builder",
|
|
"output_dir": "agents/abc12345/builder/"
|
|
}
|
|
},
|
|
"overall_success": true
|
|
}
|
|
```
|
|
|
|
## Retry Logic Architecture
|
|
|
|
### Retry Code Classification
|
|
|
|
```python
|
|
class RetryCode(str, Enum):
|
|
CLAUDE_CODE_ERROR = "claude_code_error" # Retry
|
|
TIMEOUT_ERROR = "timeout_error" # Retry
|
|
EXECUTION_ERROR = "execution_error" # Retry
|
|
ERROR_DURING_EXECUTION = "error_during_execution" # Retry
|
|
NONE = "none" # Don't retry
|
|
```
|
|
|
|
### Retry Decision Flow
|
|
|
|
```
|
|
Execute prompt
|
|
↓
|
|
Success? → Return response
|
|
↓ No
|
|
Check retry_code
|
|
↓
|
|
NONE? → Return response (don't retry)
|
|
↓ No
|
|
Retryable error?
|
|
↓ Yes
|
|
Attempts < max_retries?
|
|
↓ Yes
|
|
Wait (exponential backoff: 1s, 3s, 5s)
|
|
↓
|
|
Retry execution
|
|
↓
|
|
(Loop back to top)
|
|
↓
|
|
Max retries reached? → Return last response
|
|
```
|
|
|
|
### Default Configuration
|
|
|
|
```python
|
|
max_retries = 3
|
|
retry_delays = [1, 3, 5] # seconds
|
|
|
|
# Exponential backoff:
|
|
# Attempt 1: fail → wait 1s
|
|
# Attempt 2: fail → wait 3s
|
|
# Attempt 3: fail → wait 5s
|
|
# Attempt 4: return failure
|
|
```
|
|
|
|
## Environment Safety
|
|
|
|
### Security Model
|
|
|
|
```python
|
|
def get_safe_subprocess_env() -> Dict[str, str]:
|
|
"""Only pass essential environment variables."""
|
|
safe_vars = {
|
|
# Authentication
|
|
"ANTHROPIC_API_KEY": os.getenv("ANTHROPIC_API_KEY"),
|
|
|
|
# System essentials
|
|
"HOME": os.getenv("HOME"),
|
|
"USER": os.getenv("USER"),
|
|
"PATH": os.getenv("PATH"),
|
|
"SHELL": os.getenv("SHELL"),
|
|
|
|
# Python-specific
|
|
"PYTHONPATH": os.getenv("PYTHONPATH"),
|
|
"PYTHONUNBUFFERED": "1",
|
|
|
|
# Working directory
|
|
"PWD": os.getcwd(),
|
|
}
|
|
|
|
# Filter out None values
|
|
return {k: v for k, v in safe_vars.items() if v is not None}
|
|
```
|
|
|
|
**Why this matters**:
|
|
- Prevents leaking sensitive variables
|
|
- Subprocess isolation
|
|
- Explicit allowlist vs implicit inheritance
|
|
- Security boundary between layers
|
|
|
|
## Subprocess vs SDK: Decision Matrix
|
|
|
|
| Criteria | Subprocess (agent.py) | SDK (agent_sdk.py) |
|
|
|----------|----------------------|-------------------|
|
|
| **Type Safety** | Basic (dicts) | Strong (typed objects) |
|
|
| **Error Handling** | Generic exceptions | SDK-specific exceptions |
|
|
| **Async Support** | Subprocess management | Native async/await |
|
|
| **Dependencies** | Minimal (subprocess, json) | claude-code-sdk package |
|
|
| **Debugging** | Read JSONL files | SDK message inspection |
|
|
| **Interactive Sessions** | ❌ Not supported | ✅ ClaudeSDKClient |
|
|
| **Shell Compatibility** | ✅ Works everywhere | Python only |
|
|
| **Use Case** | Simple prompts, automation | Complex workflows, sessions |
|
|
| **Learning Curve** | Low (familiar subprocess) | Medium (SDK concepts) |
|
|
| **Performance** | Process spawn overhead | Native Python speed |
|
|
|
|
## Scalability Patterns
|
|
|
|
### Horizontal Scaling
|
|
|
|
Run multiple ADWs in parallel:
|
|
|
|
```bash
|
|
# Terminal 1
|
|
./adws/adw_chore_implement.py "feature A" &
|
|
|
|
# Terminal 2
|
|
./adws/adw_chore_implement.py "feature B" &
|
|
|
|
# Terminal 3
|
|
./adws/adw_chore_implement.py "feature C" &
|
|
|
|
# Each gets unique adw_id
|
|
# Outputs don't conflict
|
|
# Scale compute → scale features
|
|
```
|
|
|
|
### Vertical Scaling
|
|
|
|
Break complex tasks into phases:
|
|
|
|
```
|
|
Large Feature
|
|
↓
|
|
Phase 1: Architecture Design
|
|
↓
|
|
Phase 2: Core Implementation
|
|
↓
|
|
Phase 3: Testing
|
|
↓
|
|
Phase 4: Documentation
|
|
↓
|
|
Phase 5: Deployment
|
|
```
|
|
|
|
Each phase is separate ADW execution with clear handoffs.
|
|
|
|
## Key Design Decisions
|
|
|
|
### 1. JSONL as Interchange Format
|
|
- Streamable (process as it arrives)
|
|
- Line-delimited (easy to parse)
|
|
- Standard format (widely supported)
|
|
|
|
### 2. Unique ID per Execution
|
|
- Enables parallel execution
|
|
- Clear output isolation
|
|
- Audit trail maintenance
|
|
- Debugging support
|
|
|
|
### 3. Multiple Output Formats
|
|
- JSONL for streaming
|
|
- JSON for processing
|
|
- Final object for quick access
|
|
- Summary for humans
|
|
|
|
### 4. Subprocess First, SDK Optional
|
|
- Lower barrier to entry
|
|
- Subprocess is universal
|
|
- SDK adds sophistication later
|
|
- Progressive enhancement
|
|
|
|
### 5. Environment Filtering
|
|
- Security boundary
|
|
- Explicit allowlist
|
|
- Prevents accidents
|
|
- Isolation guarantee
|
|
|
|
## Extension Points
|
|
|
|
Where to extend the architecture:
|
|
|
|
1. **New ADW Scripts** (`adws/adw_*.py`)
|
|
- Add new orchestration patterns
|
|
- Implement domain-specific workflows
|
|
|
|
2. **New Slash Commands** (`.claude/commands/*.md`)
|
|
- Template new engineering patterns
|
|
- Capture team conventions
|
|
|
|
3. **New Agent Modules** (`adws/adw_modules/*.py`)
|
|
- Add state management
|
|
- Implement workflow helpers
|
|
- Add integrations
|
|
|
|
4. **Hooks** (`.claude/hooks/*.py`)
|
|
- Pre/post tool use events
|
|
- Notification systems
|
|
- Validation gates
|
|
|
|
5. **Triggers** (`adws/adw_triggers/*.py`)
|
|
- Webhook endpoints
|
|
- Cron jobs
|
|
- Event handlers
|
|
|
|
6. **Custom Subagents** (`.claude/agents/*.md`)
|
|
- Specialized agent configurations
|
|
- Domain experts
|
|
- Tool restrictions
|
|
|
|
## Performance Considerations
|
|
|
|
### Bottlenecks
|
|
|
|
1. **Claude Code Execution Time**
|
|
- Dominant factor
|
|
- Minutes per complex task
|
|
- Mitigate: Run in parallel
|
|
|
|
2. **Subprocess Spawn Overhead**
|
|
- Minimal (~100ms)
|
|
- Negligible compared to execution
|
|
- SDK slightly faster but not significant
|
|
|
|
3. **JSONL Parsing**
|
|
- Fast (JSON is efficient)
|
|
- Linear in message count
|
|
- Not a bottleneck in practice
|
|
|
|
### Optimization Strategies
|
|
|
|
1. **Parallel Execution**
|
|
- Run independent tasks concurrently
|
|
- Each gets own adw_id
|
|
- No shared state conflicts
|
|
|
|
2. **Appropriate Model Selection**
|
|
- Use Sonnet for most tasks (faster, cheaper)
|
|
- Use Opus only for complex reasoning
|
|
- 2-3x speed difference
|
|
|
|
3. **Caching**
|
|
- Claude Code has built-in caching
|
|
- Repeated prompts are faster
|
|
- Design for cache reuse
|
|
|
|
4. **Progressive Enhancement**
|
|
- Start minimal (faster setup)
|
|
- Add features as needed
|
|
- Don't over-engineer initially
|