# How to Create a Flow Schematic Specification

Flow schematics document business processes, workflows, and system flows visually and textually. They show how information moves through systems and how users interact with features.

## Quick Start

```bash
# 1. Create a new flow schematic
scripts/generate-spec.sh flow-schematic flow-001-descriptive-slug

# 2. Open and fill in the file
# (The file will be created at: docs/specs/flow-schematic/flow-001-descriptive-slug.md)

# 3. Add diagram and flow descriptions, then validate:
scripts/validate-spec.sh docs/specs/flow-schematic/flow-001-descriptive-slug.md

# 4. Fix issues and check completeness:
scripts/check-completeness.sh docs/specs/flow-schematic/flow-001-descriptive-slug.md
```

## When to Write a Flow Schematic

Use a Flow Schematic when you need to:
- Document how users interact with a feature
- Show data flow through systems
- Illustrate decision points and branches
- Document error handling paths
- Clarify complex processes
- Enable team alignment on workflow

## Research Phase

### 1. Research Related Specifications
Find what this flow represents:

```bash
# Find business requirements this flow implements
grep -r "brd" docs/specs/ --include="*.md"

# Find design documents that mention this flow
grep -r "design" docs/specs/ --include="*.md"

# Find related components or APIs
grep -r "component\|api" docs/specs/ --include="*.md"
```

### 2. Understand the User/System
- Who are the actors in this flow? (users, systems, services)
- What are they trying to accomplish?
- What information flows between actors?
- Where are the decision points?
- What happens when things go wrong?

### 3. Review Similar Flows
- How are flows documented in your organization?
- What diagramming style is used?
- What level of detail is typical?
- What's been confusing about past flows?

## Structure & Content Guide

### Title & Metadata
- **Title**: "User Export Flow", "Payment Processing Flow", etc.
- **Actor**: Primary user or system
- **Scope**: What does this flow cover?
- **Status**: Draft | Current | Legacy

### Overview Section

```markdown
# User Bulk Export Flow

## Summary
Describes the complete workflow when a user initiates a bulk data export,
including queuing, processing, file storage, and download.

**Primary Actors**: User, Export Service, Database, S3
**Scope**: From export request to download
**Current**: Yes (live in production)

## Key Steps Overview
1. User requests export (website)
2. API queues export job
3. Worker processes export
4. File stored to S3
5. User notified and downloads
```

### Flow Diagram Section

Create a visual representation:

```markdown
## Flow Diagram

### User Export Flow (ASCII Art)

```
┌─────────────┐
│   User      │
│  (Website)  │
└──────┬──────┘
       │ 1. Click Export
       ▼
┌─────────────────────────┐
│  Export API             │
│  POST /exports          │
├─────────────────────────┤
│ 2. Validate request     │
│ 3. Create export record │
│ 4. Queue job            │
└────────┬────────────────┘
         │
         ├─→ 5. Return job_id to user
         │
         ▼
┌──────────────────────┐
│ Message Queue        │
│ (Redis Bull)         │
├──────────────────────┤
│ 6. Store export job  │
└────────┬─────────────┘
         │
         ├─→ 7. Worker picks up job
         │
         ▼
┌──────────────────────────────┐
│ Export Worker                │
├──────────────────────────────┤
│ 8. Query user data           │
│ 9. Format data (CSV/JSON)    │
│ 10. Compress file            │
└────────┬─────────────────────┘
         │
         ├─→ 11. Update job status (processing)
         │
         ▼
┌──────────────────────────┐
│ AWS S3                   │
├──────────────────────────┤
│ 12. Store file           │
│ 13. Generate signed URL  │
└────────┬─────────────────┘
         │
         ├─→ 14. Send notification email to user
         │
         ▼
┌──────────────────────────┐
│ User Email               │
├──────────────────────────┤
│ 15. Click download link  │
└────────┬─────────────────┘
         │
         ├─→ 16. Browser requests file from S3
         │
         ▼
┌──────────────────────────┐
│ File Downloaded          │
└──────────────────────────┘
```

### Swimlane Diagram (Alternative Format)

```markdown
### Alternative: Swimlane Diagram

```
User          │ Frontend      │ Export API    │ Message Queue │ Worker        │ S3
              │               │               │               │               │
1. Clicks     │               │               │               │               │
   Export    ─┼──────────────→│               │               │               │
              │  2. Form Data │               │               │               │
              │               │ 3. Validate   │               │               │
              │               │ 4. Create Job│               │               │
              │               │ 5. Queue Job ─┼──────────────→│               │
              │               │               │ 6. Job Ready  │               │
              │ 7. Show Status│               │               │               │
              │ (polling)    ←┼───────────────│ (update DB)   │               │
              │               │               │               │ 8. Get Data   │
              │               │               │               │ 9. Format     │
              │               │               │               │ 10. Compress  │
              │               │               │               │ 11. Upload   ─┼──→
              │               │               │               │               │
              │ 12. Email sent│               │               │               │
              │←──────────────┼───────────────┼───────────────┤               │
              │               │               │               │               │
14. Download  │               │               │               │               │
   Starts    ─┼──────────────→│               │               │               │
              │               │               │               │               │
              │               │ 15. GET /file ┼───────────────┼──────────────→│
              │               │               │               │ 16. Return URL
              │ File Downloaded
```
```

### Step-by-Step Description Section

Document each step in detail:

```markdown
## Detailed Flow Steps

### Phase 1: Export Request

**Step 1: User Initiates Export**
- **Actor**: User
- **Action**: Clicks "Export Data" button on website
- **Input**: Export preferences (format, data types, date range)
- **Output**: Export request form submitted

**Step 2: Frontend Sends Request**
- **Actor**: Frontend/Browser
- **Action**: Submits POST request to /exports endpoint
- **Headers**: Authorization header with JWT token
- **Body**:
  ```json
  {
    "format": "csv",
    "data_types": ["users", "transactions"],
    "date_range": { "start": "2024-01-01", "end": "2024-01-31" }
  }
  ```

**Step 3: API Validates Request**
- **Actor**: Export API
- **Action**: Validate request format and parameters
- **Checks**:
  - User authenticated?
  - Valid format type?
  - Date range valid?
  - User not already processing too many exports?
- **Success**: Continue to Step 4
- **Error**: Return 400 Bad Request with error details

**Step 4: Create Export Record**
- **Actor**: Export API
- **Action**: Store export metadata in database
- **Data Stored**:
  ```sql
  INSERT INTO exports (
    id, user_id, format, data_types, status,
    created_at, updated_at
  ) VALUES (...)
  ```
- **Status**: `queued`
- **Response**: Return 201 with export_id

### Phase 2: Job Processing

**Step 5: Queue Export Job**
- **Actor**: Export API
- **Action**: Add job to Redis queue
- **Job Format**:
  ```json
  {
    "export_id": "exp_123456",
    "user_id": "usr_789012",
    "format": "csv",
    "data_types": ["users", "transactions"]
  }
  ```
- **Queue**: Bull job queue in Redis
- **TTL**: Job removed after 7 days

**Step 6: Return to User**
- **Actor**: Export API
- **Action**: Send response to frontend
- **Response**:
  ```json
  {
    "id": "exp_123456",
    "status": "queued",
    "created_at": "2024-01-15T10:00:00Z",
    "estimated_completion": "2024-01-15T10:05:00Z"
  }
  ```

### Phase 3: Data Export

**Step 7: Worker Picks Up Job**
- **Actor**: Export Worker
- **Action**: Poll Redis queue for jobs
- **Condition**: Worker checks every 100ms
- **Process**: Dequeues oldest job, marks as processing
- **Status Update**: Export marked as `processing` in database

**Step 8-10: Process Export**
- **Actor**: Export Worker
- **Actions**:
  1. Query user data from database (user table, transaction table)
  2. Validate and transform data to requested format
  3. Write to temporary file on worker disk
  4. Compress file with gzip
- **Error Handling**: If fails, retry up to 3 times with backoff

**Step 11: Upload to S3**
- **Actor**: Export Worker
- **Action**: Upload compressed file to S3
- **Filename**: `exports/exp_123456.csv.gz`
- **ACL**: Private (only accessible via signed URL)
- **Success**: Update export status to `completed` in database

### Phase 4: Notification & Download

**Step 12: Send Notification**
- **Actor**: Notification Service (triggered by export completion event)
- **Action**: Send email to user
- **Email Content**: "Your export is ready! [Click here to download]"
- **Link**: Includes signed URL (valid for 7 days)

**Step 13: User Receives Email**
- **Actor**: User
- **Action**: Receives email notification
- **Next**: Clicks download link

**Step 14-16: Download File**
- **Actor**: User browser
- **Action**: Follows download link
- **Request**: GET /exports/exp_123456/download
- **Response**: Browser initiates file download
- **File**: exp_123456.csv.gz is saved to user's computer
```

### Decision Points Section

Document branching logic:

```markdown
## Decision Points

### Decision 1: Export Format Validation
**Question**: Is the requested export format supported?
**Options**:
- ✓ CSV: Continue to data export (Step 8)
- ✓ JSON: Continue to data export (Step 8)
- ✗ Other format: Return 400 error, user selects different format

### Decision 2: User Data Available?
**Question**: Can we successfully query user data?
**Options**:
- ✓ Yes: Continue with data transformation (Step 9)
- ✗ Database error: Retry job (up to 3 times)
- ✗ User data deleted: Return "no data" message to user

### Decision 3: File Size Check
**Question**: Is the export file within size limits?
**Options**:
- ✓ < 500MB: Proceed to upload (Step 11)
- ✗ > 500MB: Return error "export too large", offer data filtering options

### Decision 4: Export Status Check (User Polling)
**Question**: Has export job completed?
**Polling**: Frontend polls GET /exports/{id} every 5 seconds
**Options**:
- `queued`: Show "Waiting to process..."
- `processing`: Show "Processing... (40%)"
- `completed`: Show download link
- `failed`: Show error message, offer retry option
- `cancelled`: Show "Export was cancelled"
```

### Error Handling Section

```markdown
## Error Handling & Recovery

### Error 1: Invalid Request Format
**Trigger**: User submits invalid format parameter
**Response Code**: 400 Bad Request
**Message**: "Invalid format. Supported: csv, json"
**Recovery**: User submits corrected request

### Error 2: Database Connection Lost During Export
**Trigger**: Worker loses connection to database while querying data
**Response Code**: (internal, no response to user)
**Recovery**: Job retried automatically (backoff: 1s, 2s, 4s)
**Max Retries**: 3 times
**If Fails After Retries**: Export marked as `failed`, user notified

### Error 3: S3 Upload Failure
**Trigger**: S3 returns 500 error
**Recovery**: Retry with exponential backoff
**Fallback**: If retries exhausted, store to local backup, retry next hour
**User Impact**: Export shows "delayed", user can check status later

### Error 4: File Too Large
**Trigger**: Export file exceeds 500MB limit
**Response Code**: 413 Payload Too Large
**Message**: "Export data exceeds 500MB. Use date filtering to reduce size."
**Recovery**: User modifies date range and resubmits

### Timeout Handling
**Job Timeout**: If export takes > 5 minutes, job is killed
**User Notification**: "Export processing took too long. Please try again."
**Logs**: Timeout recorded for analysis
**Recovery**: User can request again (usually succeeds second time)
```

### Async/Event Section

Document asynchronous aspects:

```markdown
## Asynchronous Operations

### Event: Export Created
**Trigger**: POST /exports returns 201
**Event Published**: `export.created`
**Subscribers**: Analytics service (tracks export requests)
**Payload**:
```json
{
  "export_id": "exp_123456",
  "user_id": "usr_789012",
  "format": "csv",
  "timestamp": "2024-01-15T10:00:00Z"
}
```

### Event: Export Completed
**Trigger**: Worker successfully uploads to S3
**Event Published**: `export.completed`
**Subscribers**:
- Notification service (send email)
- Analytics service (track completion)
**Payload**:
```json
{
  "export_id": "exp_123456",
  "file_size_bytes": 2048576,
  "processing_time_ms": 312000,
  "timestamp": "2024-01-15T10:05:12Z"
}
```

### Event: Export Failed
**Trigger**: Job fails after max retries
**Event Published**: `export.failed`
**Subscribers**: Notification service (alert user)
**Payload**:
```json
{
  "export_id": "exp_123456",
  "error_code": "database_timeout",
  "error_message": "Connection timeout after 3 retries",
  "timestamp": "2024-01-15T10:06:00Z"
}
```
```

### Performance & Timing Section

```markdown
## Performance Characteristics

### Typical Timings
- Request submission → queued: < 100ms
- Queued → processing starts: < 30 seconds (depends on queue load)
- Processing time:
  - Small dataset (< 10MB): 1-2 minutes
  - Medium dataset (10-100MB): 2-5 minutes
  - Large dataset (100-500MB): 5-10 minutes
- Upload to S3: 30 seconds to 2 minutes

### Total End-to-End Time
- Average: 5-10 minutes from request to download ready
- Best case: 3-5 minutes (empty queue, small dataset)
- Worst case: 15+ minutes (high load, large dataset)

### Scaling Behavior
- 1 worker: Processes 1 export at a time
- 3 workers: Process 3 exports in parallel
- 10 workers: Can handle 10 concurrent exports
- Queue depth auto-scales workers up to 20 pods
```

## Writing Tips

### Use Clear Diagrams
- ASCII art is fine and versioning-friendly
- Show all actors and their interactions
- Label arrows with what's being transmitted
- Use swimlanes for multiple actors

### Be Specific About Data
- Show actual request/response formats
- Include field names and types
- Show error responses with codes
- Document data transformations

### Cover the Happy Path AND Error Paths
- What happens when everything works?
- What happens when things go wrong?
- What are the recovery mechanisms?
- Can users recover?

### Think About Timing
- What happens asynchronously?
- Where are synchronous waits?
- What are typical timings?
- Where are bottlenecks?

### Link to Related Specs
- Reference design documents: `[DES-001]`
- Reference API contracts: `[API-001]`
- Reference component specs: `[CMP-001]`
- Reference data models: `[DATA-001]`

## Validation & Fixing Issues

### Run the Validator
```bash
scripts/validate-spec.sh docs/specs/flow-schematic/flow-001-your-spec.md
```

### Common Issues & Fixes

**Issue**: "Flow diagram incomplete or missing"
- **Fix**: Add ASCII diagram or swimlane showing all steps

**Issue**: "Step descriptions lack detail"
- **Fix**: Add what happens, who's involved, input/output for each step

**Issue**: "No error handling documented"
- **Fix**: Document error cases and recovery mechanisms

**Issue**: "Async operations not clearly shown"
- **Fix**: Highlight asynchronous steps and show event flows

## Decision-Making Framework

When documenting a flow:

1. **Scope**: What does this flow cover?
   - Where does it start/end?
   - What's in scope vs. out?

2. **Actors**: Who/what are the main actors?
   - Users, systems, services?
   - External dependencies?

3. **Happy Path**: What's the ideal flow?
   - Step-by-step happy path
   - Minimal branching

4. **Edge Cases**: What can go wrong?
   - Error scenarios
   - Recovery mechanisms
   - User impact

5. **Timing**: What's the performance profile?
   - Synchronous waits?
   - Asynchronous operations?
   - Expected timings?

## Next Steps

1. **Create the spec**: `scripts/generate-spec.sh flow-schematic flow-XXX-slug`
2. **Research**: Find related specs and understand context
3. **Sketch diagram**: Draw initial flow with all actors
4. **Document steps**: Write detailed description for each step
5. **Add error handling**: Document failure scenarios
6. **Validate**: `scripts/validate-spec.sh docs/specs/flow-schematic/flow-XXX-slug.md`
7. **Get feedback** from team to refine flow