565 lines
18 KiB
Markdown
565 lines
18 KiB
Markdown
# How to Create a Flow Schematic Specification
|
|
|
|
Flow schematics document business processes, workflows, and system flows visually and textually. They show how information moves through systems and how users interact with features.
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Create a new flow schematic
|
|
scripts/generate-spec.sh flow-schematic flow-001-descriptive-slug
|
|
|
|
# 2. Open and fill in the file
|
|
# (The file will be created at: docs/specs/flow-schematic/flow-001-descriptive-slug.md)
|
|
|
|
# 3. Add diagram and flow descriptions, then validate:
|
|
scripts/validate-spec.sh docs/specs/flow-schematic/flow-001-descriptive-slug.md
|
|
|
|
# 4. Fix issues and check completeness:
|
|
scripts/check-completeness.sh docs/specs/flow-schematic/flow-001-descriptive-slug.md
|
|
```
|
|
|
|
## When to Write a Flow Schematic
|
|
|
|
Use a Flow Schematic when you need to:
|
|
- Document how users interact with a feature
|
|
- Show data flow through systems
|
|
- Illustrate decision points and branches
|
|
- Document error handling paths
|
|
- Clarify complex processes
|
|
- Enable team alignment on workflow
|
|
|
|
## Research Phase
|
|
|
|
### 1. Research Related Specifications
|
|
Find what this flow represents:
|
|
|
|
```bash
|
|
# Find business requirements this flow implements
|
|
grep -r "brd" docs/specs/ --include="*.md"
|
|
|
|
# Find design documents that mention this flow
|
|
grep -r "design" docs/specs/ --include="*.md"
|
|
|
|
# Find related components or APIs
|
|
grep -r "component\|api" docs/specs/ --include="*.md"
|
|
```
|
|
|
|
### 2. Understand the User/System
|
|
- Who are the actors in this flow? (users, systems, services)
|
|
- What are they trying to accomplish?
|
|
- What information flows between actors?
|
|
- Where are the decision points?
|
|
- What happens when things go wrong?
|
|
|
|
### 3. Review Similar Flows
|
|
- How are flows documented in your organization?
|
|
- What diagramming style is used?
|
|
- What level of detail is typical?
|
|
- What's been confusing about past flows?
|
|
|
|
## Structure & Content Guide
|
|
|
|
### Title & Metadata
|
|
- **Title**: "User Export Flow", "Payment Processing Flow", etc.
|
|
- **Actor**: Primary user or system
|
|
- **Scope**: What does this flow cover?
|
|
- **Status**: Draft | Current | Legacy
|
|
|
|
### Overview Section
|
|
|
|
```markdown
|
|
# User Bulk Export Flow
|
|
|
|
## Summary
|
|
Describes the complete workflow when a user initiates a bulk data export,
|
|
including queuing, processing, file storage, and download.
|
|
|
|
**Primary Actors**: User, Export Service, Database, S3
|
|
**Scope**: From export request to download
|
|
**Current**: Yes (live in production)
|
|
|
|
## Key Steps Overview
|
|
1. User requests export (website)
|
|
2. API queues export job
|
|
3. Worker processes export
|
|
4. File stored to S3
|
|
5. User notified and downloads
|
|
```
|
|
|
|
### Flow Diagram Section
|
|
|
|
Create a visual representation:
|
|
|
|
```markdown
|
|
## Flow Diagram
|
|
|
|
### User Export Flow (ASCII Art)
|
|
|
|
```
|
|
┌─────────────┐
|
|
│ User │
|
|
│ (Website) │
|
|
└──────┬──────┘
|
|
│ 1. Click Export
|
|
▼
|
|
┌─────────────────────────┐
|
|
│ Export API │
|
|
│ POST /exports │
|
|
├─────────────────────────┤
|
|
│ 2. Validate request │
|
|
│ 3. Create export record │
|
|
│ 4. Queue job │
|
|
└────────┬────────────────┘
|
|
│
|
|
├─→ 5. Return job_id to user
|
|
│
|
|
▼
|
|
┌──────────────────────┐
|
|
│ Message Queue │
|
|
│ (Redis Bull) │
|
|
├──────────────────────┤
|
|
│ 6. Store export job │
|
|
└────────┬─────────────┘
|
|
│
|
|
├─→ 7. Worker picks up job
|
|
│
|
|
▼
|
|
┌──────────────────────────────┐
|
|
│ Export Worker │
|
|
├──────────────────────────────┤
|
|
│ 8. Query user data │
|
|
│ 9. Format data (CSV/JSON) │
|
|
│ 10. Compress file │
|
|
└────────┬─────────────────────┘
|
|
│
|
|
├─→ 11. Update job status (processing)
|
|
│
|
|
▼
|
|
┌──────────────────────────┐
|
|
│ AWS S3 │
|
|
├──────────────────────────┤
|
|
│ 12. Store file │
|
|
│ 13. Generate signed URL │
|
|
└────────┬─────────────────┘
|
|
│
|
|
├─→ 14. Send notification email to user
|
|
│
|
|
▼
|
|
┌──────────────────────────┐
|
|
│ User Email │
|
|
├──────────────────────────┤
|
|
│ 15. Click download link │
|
|
└────────┬─────────────────┘
|
|
│
|
|
├─→ 16. Browser requests file from S3
|
|
│
|
|
▼
|
|
┌──────────────────────────┐
|
|
│ File Downloaded │
|
|
└──────────────────────────┘
|
|
```
|
|
|
|
### Swimlane Diagram (Alternative Format)
|
|
|
|
```markdown
|
|
### Alternative: Swimlane Diagram
|
|
|
|
```
|
|
User │ Frontend │ Export API │ Message Queue │ Worker │ S3
|
|
│ │ │ │ │
|
|
1. Clicks │ │ │ │ │
|
|
Export ─┼──────────────→│ │ │ │
|
|
│ 2. Form Data │ │ │ │
|
|
│ │ 3. Validate │ │ │
|
|
│ │ 4. Create Job│ │ │
|
|
│ │ 5. Queue Job ─┼──────────────→│ │
|
|
│ │ │ 6. Job Ready │ │
|
|
│ 7. Show Status│ │ │ │
|
|
│ (polling) ←┼───────────────│ (update DB) │ │
|
|
│ │ │ │ 8. Get Data │
|
|
│ │ │ │ 9. Format │
|
|
│ │ │ │ 10. Compress │
|
|
│ │ │ │ 11. Upload ─┼──→
|
|
│ │ │ │ │
|
|
│ 12. Email sent│ │ │ │
|
|
│←──────────────┼───────────────┼───────────────┤ │
|
|
│ │ │ │ │
|
|
14. Download │ │ │ │ │
|
|
Starts ─┼──────────────→│ │ │ │
|
|
│ │ │ │ │
|
|
│ │ 15. GET /file ┼───────────────┼──────────────→│
|
|
│ │ │ │ 16. Return URL
|
|
│ File Downloaded
|
|
```
|
|
```
|
|
|
|
### Step-by-Step Description Section
|
|
|
|
Document each step in detail:
|
|
|
|
```markdown
|
|
## Detailed Flow Steps
|
|
|
|
### Phase 1: Export Request
|
|
|
|
**Step 1: User Initiates Export**
|
|
- **Actor**: User
|
|
- **Action**: Clicks "Export Data" button on website
|
|
- **Input**: Export preferences (format, data types, date range)
|
|
- **Output**: Export request form submitted
|
|
|
|
**Step 2: Frontend Sends Request**
|
|
- **Actor**: Frontend/Browser
|
|
- **Action**: Submits POST request to /exports endpoint
|
|
- **Headers**: Authorization header with JWT token
|
|
- **Body**:
|
|
```json
|
|
{
|
|
"format": "csv",
|
|
"data_types": ["users", "transactions"],
|
|
"date_range": { "start": "2024-01-01", "end": "2024-01-31" }
|
|
}
|
|
```
|
|
|
|
**Step 3: API Validates Request**
|
|
- **Actor**: Export API
|
|
- **Action**: Validate request format and parameters
|
|
- **Checks**:
|
|
- User authenticated?
|
|
- Valid format type?
|
|
- Date range valid?
|
|
- User not already processing too many exports?
|
|
- **Success**: Continue to Step 4
|
|
- **Error**: Return 400 Bad Request with error details
|
|
|
|
**Step 4: Create Export Record**
|
|
- **Actor**: Export API
|
|
- **Action**: Store export metadata in database
|
|
- **Data Stored**:
|
|
```sql
|
|
INSERT INTO exports (
|
|
id, user_id, format, data_types, status,
|
|
created_at, updated_at
|
|
) VALUES (...)
|
|
```
|
|
- **Status**: `queued`
|
|
- **Response**: Return 201 with export_id
|
|
|
|
### Phase 2: Job Processing
|
|
|
|
**Step 5: Queue Export Job**
|
|
- **Actor**: Export API
|
|
- **Action**: Add job to Redis queue
|
|
- **Job Format**:
|
|
```json
|
|
{
|
|
"export_id": "exp_123456",
|
|
"user_id": "usr_789012",
|
|
"format": "csv",
|
|
"data_types": ["users", "transactions"]
|
|
}
|
|
```
|
|
- **Queue**: Bull job queue in Redis
|
|
- **TTL**: Job removed after 7 days
|
|
|
|
**Step 6: Return to User**
|
|
- **Actor**: Export API
|
|
- **Action**: Send response to frontend
|
|
- **Response**:
|
|
```json
|
|
{
|
|
"id": "exp_123456",
|
|
"status": "queued",
|
|
"created_at": "2024-01-15T10:00:00Z",
|
|
"estimated_completion": "2024-01-15T10:05:00Z"
|
|
}
|
|
```
|
|
|
|
### Phase 3: Data Export
|
|
|
|
**Step 7: Worker Picks Up Job**
|
|
- **Actor**: Export Worker
|
|
- **Action**: Poll Redis queue for jobs
|
|
- **Condition**: Worker checks every 100ms
|
|
- **Process**: Dequeues oldest job, marks as processing
|
|
- **Status Update**: Export marked as `processing` in database
|
|
|
|
**Step 8-10: Process Export**
|
|
- **Actor**: Export Worker
|
|
- **Actions**:
|
|
1. Query user data from database (user table, transaction table)
|
|
2. Validate and transform data to requested format
|
|
3. Write to temporary file on worker disk
|
|
4. Compress file with gzip
|
|
- **Error Handling**: If fails, retry up to 3 times with backoff
|
|
|
|
**Step 11: Upload to S3**
|
|
- **Actor**: Export Worker
|
|
- **Action**: Upload compressed file to S3
|
|
- **Filename**: `exports/exp_123456.csv.gz`
|
|
- **ACL**: Private (only accessible via signed URL)
|
|
- **Success**: Update export status to `completed` in database
|
|
|
|
### Phase 4: Notification & Download
|
|
|
|
**Step 12: Send Notification**
|
|
- **Actor**: Notification Service (triggered by export completion event)
|
|
- **Action**: Send email to user
|
|
- **Email Content**: "Your export is ready! [Click here to download]"
|
|
- **Link**: Includes signed URL (valid for 7 days)
|
|
|
|
**Step 13: User Receives Email**
|
|
- **Actor**: User
|
|
- **Action**: Receives email notification
|
|
- **Next**: Clicks download link
|
|
|
|
**Step 14-16: Download File**
|
|
- **Actor**: User browser
|
|
- **Action**: Follows download link
|
|
- **Request**: GET /exports/exp_123456/download
|
|
- **Response**: Browser initiates file download
|
|
- **File**: exp_123456.csv.gz is saved to user's computer
|
|
```
|
|
|
|
### Decision Points Section
|
|
|
|
Document branching logic:
|
|
|
|
```markdown
|
|
## Decision Points
|
|
|
|
### Decision 1: Export Format Validation
|
|
**Question**: Is the requested export format supported?
|
|
**Options**:
|
|
- ✓ CSV: Continue to data export (Step 8)
|
|
- ✓ JSON: Continue to data export (Step 8)
|
|
- ✗ Other format: Return 400 error, user selects different format
|
|
|
|
### Decision 2: User Data Available?
|
|
**Question**: Can we successfully query user data?
|
|
**Options**:
|
|
- ✓ Yes: Continue with data transformation (Step 9)
|
|
- ✗ Database error: Retry job (up to 3 times)
|
|
- ✗ User data deleted: Return "no data" message to user
|
|
|
|
### Decision 3: File Size Check
|
|
**Question**: Is the export file within size limits?
|
|
**Options**:
|
|
- ✓ < 500MB: Proceed to upload (Step 11)
|
|
- ✗ > 500MB: Return error "export too large", offer data filtering options
|
|
|
|
### Decision 4: Export Status Check (User Polling)
|
|
**Question**: Has export job completed?
|
|
**Polling**: Frontend polls GET /exports/{id} every 5 seconds
|
|
**Options**:
|
|
- `queued`: Show "Waiting to process..."
|
|
- `processing`: Show "Processing... (40%)"
|
|
- `completed`: Show download link
|
|
- `failed`: Show error message, offer retry option
|
|
- `cancelled`: Show "Export was cancelled"
|
|
```
|
|
|
|
### Error Handling Section
|
|
|
|
```markdown
|
|
## Error Handling & Recovery
|
|
|
|
### Error 1: Invalid Request Format
|
|
**Trigger**: User submits invalid format parameter
|
|
**Response Code**: 400 Bad Request
|
|
**Message**: "Invalid format. Supported: csv, json"
|
|
**Recovery**: User submits corrected request
|
|
|
|
### Error 2: Database Connection Lost During Export
|
|
**Trigger**: Worker loses connection to database while querying data
|
|
**Response Code**: (internal, no response to user)
|
|
**Recovery**: Job retried automatically (backoff: 1s, 2s, 4s)
|
|
**Max Retries**: 3 times
|
|
**If Fails After Retries**: Export marked as `failed`, user notified
|
|
|
|
### Error 3: S3 Upload Failure
|
|
**Trigger**: S3 returns 500 error
|
|
**Recovery**: Retry with exponential backoff
|
|
**Fallback**: If retries exhausted, store to local backup, retry next hour
|
|
**User Impact**: Export shows "delayed", user can check status later
|
|
|
|
### Error 4: File Too Large
|
|
**Trigger**: Export file exceeds 500MB limit
|
|
**Response Code**: 413 Payload Too Large
|
|
**Message**: "Export data exceeds 500MB. Use date filtering to reduce size."
|
|
**Recovery**: User modifies date range and resubmits
|
|
|
|
### Timeout Handling
|
|
**Job Timeout**: If export takes > 5 minutes, job is killed
|
|
**User Notification**: "Export processing took too long. Please try again."
|
|
**Logs**: Timeout recorded for analysis
|
|
**Recovery**: User can request again (usually succeeds second time)
|
|
```
|
|
|
|
### Async/Event Section
|
|
|
|
Document asynchronous aspects:
|
|
|
|
```markdown
|
|
## Asynchronous Operations
|
|
|
|
### Event: Export Created
|
|
**Trigger**: POST /exports returns 201
|
|
**Event Published**: `export.created`
|
|
**Subscribers**: Analytics service (tracks export requests)
|
|
**Payload**:
|
|
```json
|
|
{
|
|
"export_id": "exp_123456",
|
|
"user_id": "usr_789012",
|
|
"format": "csv",
|
|
"timestamp": "2024-01-15T10:00:00Z"
|
|
}
|
|
```
|
|
|
|
### Event: Export Completed
|
|
**Trigger**: Worker successfully uploads to S3
|
|
**Event Published**: `export.completed`
|
|
**Subscribers**:
|
|
- Notification service (send email)
|
|
- Analytics service (track completion)
|
|
**Payload**:
|
|
```json
|
|
{
|
|
"export_id": "exp_123456",
|
|
"file_size_bytes": 2048576,
|
|
"processing_time_ms": 312000,
|
|
"timestamp": "2024-01-15T10:05:12Z"
|
|
}
|
|
```
|
|
|
|
### Event: Export Failed
|
|
**Trigger**: Job fails after max retries
|
|
**Event Published**: `export.failed`
|
|
**Subscribers**: Notification service (alert user)
|
|
**Payload**:
|
|
```json
|
|
{
|
|
"export_id": "exp_123456",
|
|
"error_code": "database_timeout",
|
|
"error_message": "Connection timeout after 3 retries",
|
|
"timestamp": "2024-01-15T10:06:00Z"
|
|
}
|
|
```
|
|
```
|
|
|
|
### Performance & Timing Section
|
|
|
|
```markdown
|
|
## Performance Characteristics
|
|
|
|
### Typical Timings
|
|
- Request submission → queued: < 100ms
|
|
- Queued → processing starts: < 30 seconds (depends on queue load)
|
|
- Processing time:
|
|
- Small dataset (< 10MB): 1-2 minutes
|
|
- Medium dataset (10-100MB): 2-5 minutes
|
|
- Large dataset (100-500MB): 5-10 minutes
|
|
- Upload to S3: 30 seconds to 2 minutes
|
|
|
|
### Total End-to-End Time
|
|
- Average: 5-10 minutes from request to download ready
|
|
- Best case: 3-5 minutes (empty queue, small dataset)
|
|
- Worst case: 15+ minutes (high load, large dataset)
|
|
|
|
### Scaling Behavior
|
|
- 1 worker: Processes 1 export at a time
|
|
- 3 workers: Process 3 exports in parallel
|
|
- 10 workers: Can handle 10 concurrent exports
|
|
- Queue depth auto-scales workers up to 20 pods
|
|
```
|
|
|
|
## Writing Tips
|
|
|
|
### Use Clear Diagrams
|
|
- ASCII art is fine and versioning-friendly
|
|
- Show all actors and their interactions
|
|
- Label arrows with what's being transmitted
|
|
- Use swimlanes for multiple actors
|
|
|
|
### Be Specific About Data
|
|
- Show actual request/response formats
|
|
- Include field names and types
|
|
- Show error responses with codes
|
|
- Document data transformations
|
|
|
|
### Cover the Happy Path AND Error Paths
|
|
- What happens when everything works?
|
|
- What happens when things go wrong?
|
|
- What are the recovery mechanisms?
|
|
- Can users recover?
|
|
|
|
### Think About Timing
|
|
- What happens asynchronously?
|
|
- Where are synchronous waits?
|
|
- What are typical timings?
|
|
- Where are bottlenecks?
|
|
|
|
### Link to Related Specs
|
|
- Reference design documents: `[DES-001]`
|
|
- Reference API contracts: `[API-001]`
|
|
- Reference component specs: `[CMP-001]`
|
|
- Reference data models: `[DATA-001]`
|
|
|
|
## Validation & Fixing Issues
|
|
|
|
### Run the Validator
|
|
```bash
|
|
scripts/validate-spec.sh docs/specs/flow-schematic/flow-001-your-spec.md
|
|
```
|
|
|
|
### Common Issues & Fixes
|
|
|
|
**Issue**: "Flow diagram incomplete or missing"
|
|
- **Fix**: Add ASCII diagram or swimlane showing all steps
|
|
|
|
**Issue**: "Step descriptions lack detail"
|
|
- **Fix**: Add what happens, who's involved, input/output for each step
|
|
|
|
**Issue**: "No error handling documented"
|
|
- **Fix**: Document error cases and recovery mechanisms
|
|
|
|
**Issue**: "Async operations not clearly shown"
|
|
- **Fix**: Highlight asynchronous steps and show event flows
|
|
|
|
## Decision-Making Framework
|
|
|
|
When documenting a flow:
|
|
|
|
1. **Scope**: What does this flow cover?
|
|
- Where does it start/end?
|
|
- What's in scope vs. out?
|
|
|
|
2. **Actors**: Who/what are the main actors?
|
|
- Users, systems, services?
|
|
- External dependencies?
|
|
|
|
3. **Happy Path**: What's the ideal flow?
|
|
- Step-by-step happy path
|
|
- Minimal branching
|
|
|
|
4. **Edge Cases**: What can go wrong?
|
|
- Error scenarios
|
|
- Recovery mechanisms
|
|
- User impact
|
|
|
|
5. **Timing**: What's the performance profile?
|
|
- Synchronous waits?
|
|
- Asynchronous operations?
|
|
- Expected timings?
|
|
|
|
## Next Steps
|
|
|
|
1. **Create the spec**: `scripts/generate-spec.sh flow-schematic flow-XXX-slug`
|
|
2. **Research**: Find related specs and understand context
|
|
3. **Sketch diagram**: Draw initial flow with all actors
|
|
4. **Document steps**: Write detailed description for each step
|
|
5. **Add error handling**: Document failure scenarios
|
|
6. **Validate**: `scripts/validate-spec.sh docs/specs/flow-schematic/flow-XXX-slug.md`
|
|
7. **Get feedback** from team to refine flow
|