Initial commit

This commit is contained in:
Zhongwei Li
2025-11-30 08:28:42 +08:00
commit 8a4be47b6e
43 changed files with 10867 additions and 0 deletions

View File

@@ -0,0 +1,687 @@
# ADW Architecture Deep Dive
## System Overview
ADW infrastructure creates a **programmatic interface** for AI-driven development by wrapping the application layer with an agentic layer that templates engineering patterns.
```
┌─────────────────────────────────────────────────────────────┐
│ Agentic Layer │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ ADW Scripts (adws/) │ │
│ │ - Execute prompts programmatically │ │
│ │ - Orchestrate multi-phase workflows │ │
│ │ - Invoke Claude Code CLI or SDK │ │
│ └───────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Slash Commands (.claude/commands/) │ │
│ │ - Reusable prompt templates │ │
│ │ - Structured instructions for agents │ │
│ │ - Variable substitution │ │
│ └───────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Specifications (specs/) │ │
│ │ - Implementation plans │ │
│ │ - Step-by-step tasks │ │
│ │ - Validation commands │ │
│ └───────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Claude Code (subprocess or SDK) │ │
│ │ - Executes prompts with tools │ │
│ │ - Modifies application code │ │
│ │ - Returns structured results │ │
│ └───────────────────────────────────────────────────────┘ │
│ ↓ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ Observability (agents/) │ │
│ │ - Structured outputs │ │
│ │ - Execution tracking │ │
│ │ - Debug artifacts │ │
│ └───────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Application Layer │
│ - Your actual application code (apps/, src/, lib/) │
│ - Tests, configs, documentation │
│ - What agents read and modify │
└─────────────────────────────────────────────────────────────┘
```
## Core Modules
### agent.py - Subprocess Execution Engine
**Purpose**: Core module for executing Claude Code CLI as subprocess
**Key Components**:
```python
# Data Models
class AgentPromptRequest(BaseModel):
prompt: str # What to execute
adw_id: str # Unique workflow ID
agent_name: str # Agent identifier
model: Literal["sonnet", "opus"]
output_file: str # Where to save output
working_dir: Optional[str] # Where to run
class AgentPromptResponse(BaseModel):
output: str # Result text
success: bool # Did it succeed?
session_id: Optional[str] # Claude session ID
retry_code: RetryCode # Should we retry?
# Core Functions
def prompt_claude_code(request: AgentPromptRequest) -> AgentPromptResponse
def prompt_claude_code_with_retry(request: AgentPromptRequest) -> AgentPromptResponse
def execute_template(request: AgentTemplateRequest) -> AgentPromptResponse
```
**Execution Flow**:
1. **Build Command**
```python
cmd = [
"claude",
"-p", prompt,
"--model", model,
"--output-format", "stream-json",
"--verbose"
]
```
2. **Filter Environment**
```python
env = get_safe_subprocess_env()
# Only passes essential variables
# Prevents environment leakage
```
3. **Execute & Stream**
```python
subprocess.run(
cmd,
stdout=output_file, # Stream to file
stderr=subprocess.PIPE,
env=env
)
```
4. **Parse JSONL Output**
```python
messages = [json.loads(line) for line in f]
result_message = find_result_message(messages)
```
5. **Convert to Multiple Formats**
- `cc_raw_output.jsonl` - Raw streaming JSONL
- `cc_raw_output.json` - Parsed JSON array
- `cc_final_object.json` - Final result object
6. **Return Response**
```python
return AgentPromptResponse(
output=result_text,
success=not is_error,
session_id=session_id,
retry_code=RetryCode.NONE
)
```
**Error Handling**:
- Retry codes distinguish transient from permanent errors
- Output truncation prevents console flooding
- Graceful degradation on parse failures
### agent_sdk.py - SDK Execution Engine
**Purpose**: Type-safe, async execution using Claude Code Python SDK
**Key Patterns**:
```python
# One-shot query
async def simple_query(prompt: str) -> str:
options = ClaudeCodeOptions(model="claude-sonnet-4-20250514")
texts = []
async for message in query(prompt=prompt, options=options):
if isinstance(message, AssistantMessage):
texts.append(extract_text(message))
return "\n".join(texts)
# Interactive session
@asynccontextmanager
async def create_session():
client = ClaudeSDKClient(options=options)
await client.connect()
try:
yield client
finally:
await client.disconnect()
```
**Advantages**:
- Native async/await patterns
- Typed message objects (AssistantMessage, ResultMessage)
- SDK-specific error handling
- Interactive session support
- Better IDE integration
**When to Use**:
- Interactive multi-turn conversations
- Need better type safety
- Async workflow integration
- Native Python integration preferred
## Data Flow Patterns
### Pattern 1: Direct Prompt Execution
```
User Input
./adws/adw_prompt.py "analyze this"
AgentPromptRequest created
prompt_claude_code_with_retry()
subprocess: claude -p "analyze this"
JSONL output streamed to file
Parse JSONL → JSON conversion
AgentPromptResponse returned
Display to user + save summary
```
### Pattern 2: Slash Command Execution
```
User Input
./adws/adw_slash_command.py /chore abc123 "task"
Read .claude/commands/chore.md
Substitute variables ($1, $2)
AgentTemplateRequest created
execute_template() → prompt_claude_code_with_retry()
subprocess: claude -p "<expanded template>"
Parse results
Extract spec path from output
Display + save
```
### Pattern 3: Compound Workflow
```
User Input
./adws/adw_chore_implement.py "feature X"
Generate unique adw_id
┌─────────────────────────────────────┐
│ Phase 1: Planning │
├─────────────────────────────────────┤
│ execute_template("/chore", args) │
│ ↓ │
│ Create plan in specs/ │
│ ↓ │
│ Parse output for plan path │
└─────────────────────────────────────┘
┌─────────────────────────────────────┐
│ Phase 2: Implementation │
├─────────────────────────────────────┤
│ execute_template("/implement", path) │
│ ↓ │
│ Read plan from specs/ │
│ ↓ │
│ Execute step by step │
│ ↓ │
│ Modify application code │
└─────────────────────────────────────┘
Workflow summary saved
Display results to user
```
## Directory Structure
### Minimal Setup
```
project/
├── adws/
│ ├── adw_modules/
│ │ └── agent.py # Core execution
│ └── adw_prompt.py # CLI wrapper
├── .claude/commands/
│ ├── chore.md # Planning template
│ └── implement.md # Implementation template
├── specs/
│ └── *.md # Generated plans
├── agents/ # Output directory
│ └── {adw_id}/
│ └── {agent_name}/
│ ├── cc_raw_output.jsonl
│ ├── cc_raw_output.json
│ ├── cc_final_object.json
│ └── custom_summary_output.json
└── .env.sample # Configuration template
```
### Enhanced Setup
```
project/
├── adws/
│ ├── adw_modules/
│ │ ├── agent.py # Subprocess execution
│ │ └── agent_sdk.py # SDK execution
│ ├── adw_prompt.py # Direct prompts
│ ├── adw_slash_command.py # Command executor
│ └── adw_chore_implement.py # Compound workflow
├── .claude/commands/
│ ├── chore.md
│ ├── implement.md
│ ├── feature.md # Feature planning
│ ├── test.md # Test creation
│ ├── prime.md # Context loading
│ └── start.md # App startup
├── specs/
│ ├── chore-*.md
│ └── feature-*.md
├── agents/
│ └── {adw_id}/
│ ├── planner/ # Planning agent outputs
│ ├── builder/ # Building agent outputs
│ └── workflow_summary.json # Overall summary
└── CLAUDE.md # Updated with ADW docs
```
### Scaled Setup (Production)
```
project/
├── adws/
│ ├── adw_modules/
│ │ ├── agent.py # Subprocess execution
│ │ ├── agent_sdk.py # SDK execution
│ │ ├── data_types.py # Type definitions
│ │ ├── state.py # State management (adw_state.json)
│ │ ├── git_ops.py # Git operations (branch, commit, push, PR)
│ │ ├── worktree_ops.py # Worktree isolation management
│ │ ├── workflow_ops.py # High-level orchestration
│ │ ├── github.py # GitHub integration (gh CLI)
│ │ └── utils.py # Shared utilities
│ ├── adw_sdlc_iso.py # Complete SDLC (plan→build→test→review→doc)
│ ├── adw_plan_build_test_review_iso.py # Compound isolated workflow
│ ├── adw_ship_iso.py # Merge validation and shipping
│ ├── adw_plan_iso.py # Isolated planning phase
│ ├── adw_build_iso.py # Isolated build phase
│ ├── adw_test_iso.py # Isolated testing phase
│ ├── adw_review_iso.py # Isolated review phase
│ └── adw_document_iso.py # Isolated documentation phase
├── .claude/commands/
│ ├── chore.md
│ ├── bug.md
│ ├── feature.md
│ ├── implement.md
│ ├── classify_issue.md # Issue classification
│ ├── classify_adw.md # ADW workflow selection
│ ├── generate_branch_name.md # Branch naming
│ ├── patch.md # Patch planning
│ ├── test.md # Test execution
│ ├── review.md # Code review
│ ├── document.md # Documentation generation
│ ├── pull_request.md # PR creation
│ ├── install_worktree.md # Worktree environment setup
│ ├── cleanup_worktrees.md # Worktree cleanup
│ ├── commit.md # Commit message generation
│ ├── prime.md
│ └── start.md
├── specs/
│ ├── chore-*.md
│ ├── bug-*.md
│ ├── feature-*.md
│ └── patch/ # Patch plans
├── agents/
│ └── {adw_id}/
│ ├── {agent_name}/ # Per-agent outputs
│ ├── adw_state.json # Persistent state
│ └── workflow_summary.json
├── trees/ # Git worktree isolation
│ └── {adw_id}/ # Isolated working directory
│ ├── .ports.env # Port configuration
│ └── <full repo copy>
├── .adw_backups/ # Safety backups during upgrades
├── CLAUDE.md # Updated with ADW docs
└── .env.sample # Configuration
```
**Key Scaled Phase Features:**
1. **State Management**: Persistent state across workflow phases via `adw_state.json`
2. **Git Worktree Isolation**: Each ADW runs in isolated `trees/{adw_id}/` directory
3. **Port Management**: Deterministic port allocation (9100-9114 backend, 9200-9214 frontend)
4. **GitHub Integration**: Issue operations, PR management, comment posting via gh CLI
5. **Multi-Phase Workflows**: Complete SDLC automation (plan, build, test, review, document, ship)
6. **Workflow Composition**: High-level functions for issue classification, branch generation, etc.
7. **Advanced Commands**: Rich library of 20+ specialized slash commands
## Output Structure
### Observability Artifacts
Each ADW execution creates:
**cc_raw_output.jsonl**
- Line-delimited JSON
- Streaming output from Claude Code
- Each line is a message object
- Last line is result message
**cc_raw_output.json**
- JSON array of all messages
- Easier for programmatic processing
- Contains full conversation history
**cc_final_object.json**
- Just the last message (result)
- Quick access to final output
- Contains success/failure info
**custom_summary_output.json**
- High-level execution summary
- Metadata (adw_id, prompt, model)
- Success status
- Session ID for debugging
### Workflow Tracking
For compound workflows:
**workflow_summary.json**
```json
{
"workflow": "chore_implement",
"adw_id": "abc12345",
"prompt": "add error handling",
"phases": {
"planning": {
"success": true,
"session_id": "session_xyz",
"agent": "planner",
"output_dir": "agents/abc12345/planner/"
},
"implementation": {
"success": true,
"session_id": "session_abc",
"agent": "builder",
"output_dir": "agents/abc12345/builder/"
}
},
"overall_success": true
}
```
## Retry Logic Architecture
### Retry Code Classification
```python
class RetryCode(str, Enum):
CLAUDE_CODE_ERROR = "claude_code_error" # Retry
TIMEOUT_ERROR = "timeout_error" # Retry
EXECUTION_ERROR = "execution_error" # Retry
ERROR_DURING_EXECUTION = "error_during_execution" # Retry
NONE = "none" # Don't retry
```
### Retry Decision Flow
```
Execute prompt
Success? → Return response
↓ No
Check retry_code
NONE? → Return response (don't retry)
↓ No
Retryable error?
↓ Yes
Attempts < max_retries?
↓ Yes
Wait (exponential backoff: 1s, 3s, 5s)
Retry execution
(Loop back to top)
Max retries reached? → Return last response
```
### Default Configuration
```python
max_retries = 3
retry_delays = [1, 3, 5] # seconds
# Exponential backoff:
# Attempt 1: fail → wait 1s
# Attempt 2: fail → wait 3s
# Attempt 3: fail → wait 5s
# Attempt 4: return failure
```
## Environment Safety
### Security Model
```python
def get_safe_subprocess_env() -> Dict[str, str]:
"""Only pass essential environment variables."""
safe_vars = {
# Authentication
"ANTHROPIC_API_KEY": os.getenv("ANTHROPIC_API_KEY"),
# System essentials
"HOME": os.getenv("HOME"),
"USER": os.getenv("USER"),
"PATH": os.getenv("PATH"),
"SHELL": os.getenv("SHELL"),
# Python-specific
"PYTHONPATH": os.getenv("PYTHONPATH"),
"PYTHONUNBUFFERED": "1",
# Working directory
"PWD": os.getcwd(),
}
# Filter out None values
return {k: v for k, v in safe_vars.items() if v is not None}
```
**Why this matters**:
- Prevents leaking sensitive variables
- Subprocess isolation
- Explicit allowlist vs implicit inheritance
- Security boundary between layers
## Subprocess vs SDK: Decision Matrix
| Criteria | Subprocess (agent.py) | SDK (agent_sdk.py) |
|----------|----------------------|-------------------|
| **Type Safety** | Basic (dicts) | Strong (typed objects) |
| **Error Handling** | Generic exceptions | SDK-specific exceptions |
| **Async Support** | Subprocess management | Native async/await |
| **Dependencies** | Minimal (subprocess, json) | claude-code-sdk package |
| **Debugging** | Read JSONL files | SDK message inspection |
| **Interactive Sessions** | ❌ Not supported | ✅ ClaudeSDKClient |
| **Shell Compatibility** | ✅ Works everywhere | Python only |
| **Use Case** | Simple prompts, automation | Complex workflows, sessions |
| **Learning Curve** | Low (familiar subprocess) | Medium (SDK concepts) |
| **Performance** | Process spawn overhead | Native Python speed |
## Scalability Patterns
### Horizontal Scaling
Run multiple ADWs in parallel:
```bash
# Terminal 1
./adws/adw_chore_implement.py "feature A" &
# Terminal 2
./adws/adw_chore_implement.py "feature B" &
# Terminal 3
./adws/adw_chore_implement.py "feature C" &
# Each gets unique adw_id
# Outputs don't conflict
# Scale compute → scale features
```
### Vertical Scaling
Break complex tasks into phases:
```
Large Feature
Phase 1: Architecture Design
Phase 2: Core Implementation
Phase 3: Testing
Phase 4: Documentation
Phase 5: Deployment
```
Each phase is separate ADW execution with clear handoffs.
## Key Design Decisions
### 1. JSONL as Interchange Format
- Streamable (process as it arrives)
- Line-delimited (easy to parse)
- Standard format (widely supported)
### 2. Unique ID per Execution
- Enables parallel execution
- Clear output isolation
- Audit trail maintenance
- Debugging support
### 3. Multiple Output Formats
- JSONL for streaming
- JSON for processing
- Final object for quick access
- Summary for humans
### 4. Subprocess First, SDK Optional
- Lower barrier to entry
- Subprocess is universal
- SDK adds sophistication later
- Progressive enhancement
### 5. Environment Filtering
- Security boundary
- Explicit allowlist
- Prevents accidents
- Isolation guarantee
## Extension Points
Where to extend the architecture:
1. **New ADW Scripts** (`adws/adw_*.py`)
- Add new orchestration patterns
- Implement domain-specific workflows
2. **New Slash Commands** (`.claude/commands/*.md`)
- Template new engineering patterns
- Capture team conventions
3. **New Agent Modules** (`adws/adw_modules/*.py`)
- Add state management
- Implement workflow helpers
- Add integrations
4. **Hooks** (`.claude/hooks/*.py`)
- Pre/post tool use events
- Notification systems
- Validation gates
5. **Triggers** (`adws/adw_triggers/*.py`)
- Webhook endpoints
- Cron jobs
- Event handlers
6. **Custom Subagents** (`.claude/agents/*.md`)
- Specialized agent configurations
- Domain experts
- Tool restrictions
## Performance Considerations
### Bottlenecks
1. **Claude Code Execution Time**
- Dominant factor
- Minutes per complex task
- Mitigate: Run in parallel
2. **Subprocess Spawn Overhead**
- Minimal (~100ms)
- Negligible compared to execution
- SDK slightly faster but not significant
3. **JSONL Parsing**
- Fast (JSON is efficient)
- Linear in message count
- Not a bottleneck in practice
### Optimization Strategies
1. **Parallel Execution**
- Run independent tasks concurrently
- Each gets own adw_id
- No shared state conflicts
2. **Appropriate Model Selection**
- Use Sonnet for most tasks (faster, cheaper)
- Use Opus only for complex reasoning
- 2-3x speed difference
3. **Caching**
- Claude Code has built-in caching
- Repeated prompts are faster
- Design for cache reuse
4. **Progressive Enhancement**
- Start minimal (faster setup)
- Add features as needed
- Don't over-engineer initially