Initial commit
This commit is contained in:
Zhongwei Li
564
skills/spec-author/guides/flow-schematic.md
Normal file
564
skills/spec-author/guides/flow-schematic.md
Normal file
@@ -0,0 +1,564 @@
|
||||
# How to Create a Flow Schematic Specification
|
||||
|
||||
Flow schematics document business processes, workflows, and system flows visually and textually. They show how information moves through systems and how users interact with features.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Create a new flow schematic
|
||||
scripts/generate-spec.sh flow-schematic flow-001-descriptive-slug
|
||||
|
||||
# 2. Open and fill in the file
|
||||
# (The file will be created at: docs/specs/flow-schematic/flow-001-descriptive-slug.md)
|
||||
|
||||
# 3. Add diagram and flow descriptions, then validate:
|
||||
scripts/validate-spec.sh docs/specs/flow-schematic/flow-001-descriptive-slug.md
|
||||
|
||||
# 4. Fix issues and check completeness:
|
||||
scripts/check-completeness.sh docs/specs/flow-schematic/flow-001-descriptive-slug.md
|
||||
```
|
||||
|
||||
## When to Write a Flow Schematic
|
||||
|
||||
Use a Flow Schematic when you need to:
|
||||
- Document how users interact with a feature
|
||||
- Show data flow through systems
|
||||
- Illustrate decision points and branches
|
||||
- Document error handling paths
|
||||
- Clarify complex processes
|
||||
- Enable team alignment on workflow
|
||||
|
||||
## Research Phase
|
||||
|
||||
### 1. Research Related Specifications
|
||||
Find what this flow represents:
|
||||
|
||||
```bash
|
||||
# Find business requirements this flow implements
|
||||
grep -r "brd" docs/specs/ --include="*.md"
|
||||
|
||||
# Find design documents that mention this flow
|
||||
grep -r "design" docs/specs/ --include="*.md"
|
||||
|
||||
# Find related components or APIs
|
||||
grep -r "component\|api" docs/specs/ --include="*.md"
|
||||
```
|
||||
|
||||
### 2. Understand the User/System
|
||||
- Who are the actors in this flow? (users, systems, services)
|
||||
- What are they trying to accomplish?
|
||||
- What information flows between actors?
|
||||
- Where are the decision points?
|
||||
- What happens when things go wrong?
|
||||
|
||||
### 3. Review Similar Flows
|
||||
- How are flows documented in your organization?
|
||||
- What diagramming style is used?
|
||||
- What level of detail is typical?
|
||||
- What's been confusing about past flows?
|
||||
|
||||
## Structure & Content Guide
|
||||
|
||||
### Title & Metadata
|
||||
- **Title**: "User Export Flow", "Payment Processing Flow", etc.
|
||||
- **Actor**: Primary user or system
|
||||
- **Scope**: What does this flow cover?
|
||||
- **Status**: Draft | Current | Legacy
|
||||
|
||||
### Overview Section
|
||||
|
||||
```markdown
|
||||
# User Bulk Export Flow
|
||||
|
||||
## Summary
|
||||
Describes the complete workflow when a user initiates a bulk data export,
|
||||
including queuing, processing, file storage, and download.
|
||||
|
||||
**Primary Actors**: User, Export Service, Database, S3
|
||||
**Scope**: From export request to download
|
||||
**Current**: Yes (live in production)
|
||||
|
||||
## Key Steps Overview
|
||||
1. User requests export (website)
|
||||
2. API queues export job
|
||||
3. Worker processes export
|
||||
4. File stored to S3
|
||||
5. User notified and downloads
|
||||
```
|
||||
|
||||
### Flow Diagram Section
|
||||
|
||||
Create a visual representation:
|
||||
|
||||
```markdown
|
||||
## Flow Diagram
|
||||
|
||||
### User Export Flow (ASCII Art)
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ User │
|
||||
│ (Website) │
|
||||
└──────┬──────┘
|
||||
│ 1. Click Export
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ Export API │
|
||||
│ POST /exports │
|
||||
├─────────────────────────┤
|
||||
│ 2. Validate request │
|
||||
│ 3. Create export record │
|
||||
│ 4. Queue job │
|
||||
└────────┬────────────────┘
|
||||
│
|
||||
├─→ 5. Return job_id to user
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Message Queue │
|
||||
│ (Redis Bull) │
|
||||
├──────────────────────┤
|
||||
│ 6. Store export job │
|
||||
└────────┬─────────────┘
|
||||
│
|
||||
├─→ 7. Worker picks up job
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────┐
|
||||
│ Export Worker │
|
||||
├──────────────────────────────┤
|
||||
│ 8. Query user data │
|
||||
│ 9. Format data (CSV/JSON) │
|
||||
│ 10. Compress file │
|
||||
└────────┬─────────────────────┘
|
||||
│
|
||||
├─→ 11. Update job status (processing)
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ AWS S3 │
|
||||
├──────────────────────────┤
|
||||
│ 12. Store file │
|
||||
│ 13. Generate signed URL │
|
||||
└────────┬─────────────────┘
|
||||
│
|
||||
├─→ 14. Send notification email to user
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ User Email │
|
||||
├──────────────────────────┤
|
||||
│ 15. Click download link │
|
||||
└────────┬─────────────────┘
|
||||
│
|
||||
├─→ 16. Browser requests file from S3
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────┐
|
||||
│ File Downloaded │
|
||||
└──────────────────────────┘
|
||||
```
|
||||
|
||||
### Swimlane Diagram (Alternative Format)
|
||||
|
||||
```markdown
|
||||
### Alternative: Swimlane Diagram
|
||||
|
||||
```
|
||||
User │ Frontend │ Export API │ Message Queue │ Worker │ S3
|
||||
│ │ │ │ │
|
||||
1. Clicks │ │ │ │ │
|
||||
Export ─┼──────────────→│ │ │ │
|
||||
│ 2. Form Data │ │ │ │
|
||||
│ │ 3. Validate │ │ │
|
||||
│ │ 4. Create Job│ │ │
|
||||
│ │ 5. Queue Job ─┼──────────────→│ │
|
||||
│ │ │ 6. Job Ready │ │
|
||||
│ 7. Show Status│ │ │ │
|
||||
│ (polling) ←┼───────────────│ (update DB) │ │
|
||||
│ │ │ │ 8. Get Data │
|
||||
│ │ │ │ 9. Format │
|
||||
│ │ │ │ 10. Compress │
|
||||
│ │ │ │ 11. Upload ─┼──→
|
||||
│ │ │ │ │
|
||||
│ 12. Email sent│ │ │ │
|
||||
│←──────────────┼───────────────┼───────────────┤ │
|
||||
│ │ │ │ │
|
||||
14. Download │ │ │ │ │
|
||||
Starts ─┼──────────────→│ │ │ │
|
||||
│ │ │ │ │
|
||||
│ │ 15. GET /file ┼───────────────┼──────────────→│
|
||||
│ │ │ │ 16. Return URL
|
||||
│ File Downloaded
|
||||
```
|
||||
```
|
||||
|
||||
### Step-by-Step Description Section
|
||||
|
||||
Document each step in detail:
|
||||
|
||||
```markdown
|
||||
## Detailed Flow Steps
|
||||
|
||||
### Phase 1: Export Request
|
||||
|
||||
**Step 1: User Initiates Export**
|
||||
- **Actor**: User
|
||||
- **Action**: Clicks "Export Data" button on website
|
||||
- **Input**: Export preferences (format, data types, date range)
|
||||
- **Output**: Export request form submitted
|
||||
|
||||
**Step 2: Frontend Sends Request**
|
||||
- **Actor**: Frontend/Browser
|
||||
- **Action**: Submits POST request to /exports endpoint
|
||||
- **Headers**: Authorization header with JWT token
|
||||
- **Body**:
|
||||
```json
|
||||
{
|
||||
"format": "csv",
|
||||
"data_types": ["users", "transactions"],
|
||||
"date_range": { "start": "2024-01-01", "end": "2024-01-31" }
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3: API Validates Request**
|
||||
- **Actor**: Export API
|
||||
- **Action**: Validate request format and parameters
|
||||
- **Checks**:
|
||||
- User authenticated?
|
||||
- Valid format type?
|
||||
- Date range valid?
|
||||
- User not already processing too many exports?
|
||||
- **Success**: Continue to Step 4
|
||||
- **Error**: Return 400 Bad Request with error details
|
||||
|
||||
**Step 4: Create Export Record**
|
||||
- **Actor**: Export API
|
||||
- **Action**: Store export metadata in database
|
||||
- **Data Stored**:
|
||||
```sql
|
||||
INSERT INTO exports (
|
||||
id, user_id, format, data_types, status,
|
||||
created_at, updated_at
|
||||
) VALUES (...)
|
||||
```
|
||||
- **Status**: `queued`
|
||||
- **Response**: Return 201 with export_id
|
||||
|
||||
### Phase 2: Job Processing
|
||||
|
||||
**Step 5: Queue Export Job**
|
||||
- **Actor**: Export API
|
||||
- **Action**: Add job to Redis queue
|
||||
- **Job Format**:
|
||||
```json
|
||||
{
|
||||
"export_id": "exp_123456",
|
||||
"user_id": "usr_789012",
|
||||
"format": "csv",
|
||||
"data_types": ["users", "transactions"]
|
||||
}
|
||||
```
|
||||
- **Queue**: Bull job queue in Redis
|
||||
- **TTL**: Job removed after 7 days
|
||||
|
||||
**Step 6: Return to User**
|
||||
- **Actor**: Export API
|
||||
- **Action**: Send response to frontend
|
||||
- **Response**:
|
||||
```json
|
||||
{
|
||||
"id": "exp_123456",
|
||||
"status": "queued",
|
||||
"created_at": "2024-01-15T10:00:00Z",
|
||||
"estimated_completion": "2024-01-15T10:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 3: Data Export
|
||||
|
||||
**Step 7: Worker Picks Up Job**
|
||||
- **Actor**: Export Worker
|
||||
- **Action**: Poll Redis queue for jobs
|
||||
- **Condition**: Worker checks every 100ms
|
||||
- **Process**: Dequeues oldest job, marks as processing
|
||||
- **Status Update**: Export marked as `processing` in database
|
||||
|
||||
**Step 8-10: Process Export**
|
||||
- **Actor**: Export Worker
|
||||
- **Actions**:
|
||||
1. Query user data from database (user table, transaction table)
|
||||
2. Validate and transform data to requested format
|
||||
3. Write to temporary file on worker disk
|
||||
4. Compress file with gzip
|
||||
- **Error Handling**: If fails, retry up to 3 times with backoff
|
||||
|
||||
**Step 11: Upload to S3**
|
||||
- **Actor**: Export Worker
|
||||
- **Action**: Upload compressed file to S3
|
||||
- **Filename**: `exports/exp_123456.csv.gz`
|
||||
- **ACL**: Private (only accessible via signed URL)
|
||||
- **Success**: Update export status to `completed` in database
|
||||
|
||||
### Phase 4: Notification & Download
|
||||
|
||||
**Step 12: Send Notification**
|
||||
- **Actor**: Notification Service (triggered by export completion event)
|
||||
- **Action**: Send email to user
|
||||
- **Email Content**: "Your export is ready! [Click here to download]"
|
||||
- **Link**: Includes signed URL (valid for 7 days)
|
||||
|
||||
**Step 13: User Receives Email**
|
||||
- **Actor**: User
|
||||
- **Action**: Receives email notification
|
||||
- **Next**: Clicks download link
|
||||
|
||||
**Step 14-16: Download File**
|
||||
- **Actor**: User browser
|
||||
- **Action**: Follows download link
|
||||
- **Request**: GET /exports/exp_123456/download
|
||||
- **Response**: Browser initiates file download
|
||||
- **File**: exp_123456.csv.gz is saved to user's computer
|
||||
```
|
||||
|
||||
### Decision Points Section
|
||||
|
||||
Document branching logic:
|
||||
|
||||
```markdown
|
||||
## Decision Points
|
||||
|
||||
### Decision 1: Export Format Validation
|
||||
**Question**: Is the requested export format supported?
|
||||
**Options**:
|
||||
- ✓ CSV: Continue to data export (Step 8)
|
||||
- ✓ JSON: Continue to data export (Step 8)
|
||||
- ✗ Other format: Return 400 error, user selects different format
|
||||
|
||||
### Decision 2: User Data Available?
|
||||
**Question**: Can we successfully query user data?
|
||||
**Options**:
|
||||
- ✓ Yes: Continue with data transformation (Step 9)
|
||||
- ✗ Database error: Retry job (up to 3 times)
|
||||
- ✗ User data deleted: Return "no data" message to user
|
||||
|
||||
### Decision 3: File Size Check
|
||||
**Question**: Is the export file within size limits?
|
||||
**Options**:
|
||||
- ✓ < 500MB: Proceed to upload (Step 11)
|
||||
- ✗ > 500MB: Return error "export too large", offer data filtering options
|
||||
|
||||
### Decision 4: Export Status Check (User Polling)
|
||||
**Question**: Has export job completed?
|
||||
**Polling**: Frontend polls GET /exports/{id} every 5 seconds
|
||||
**Options**:
|
||||
- `queued`: Show "Waiting to process..."
|
||||
- `processing`: Show "Processing... (40%)"
|
||||
- `completed`: Show download link
|
||||
- `failed`: Show error message, offer retry option
|
||||
- `cancelled`: Show "Export was cancelled"
|
||||
```
|
||||
|
||||
### Error Handling Section
|
||||
|
||||
```markdown
|
||||
## Error Handling & Recovery
|
||||
|
||||
### Error 1: Invalid Request Format
|
||||
**Trigger**: User submits invalid format parameter
|
||||
**Response Code**: 400 Bad Request
|
||||
**Message**: "Invalid format. Supported: csv, json"
|
||||
**Recovery**: User submits corrected request
|
||||
|
||||
### Error 2: Database Connection Lost During Export
|
||||
**Trigger**: Worker loses connection to database while querying data
|
||||
**Response Code**: (internal, no response to user)
|
||||
**Recovery**: Job retried automatically (backoff: 1s, 2s, 4s)
|
||||
**Max Retries**: 3 times
|
||||
**If Fails After Retries**: Export marked as `failed`, user notified
|
||||
|
||||
### Error 3: S3 Upload Failure
|
||||
**Trigger**: S3 returns 500 error
|
||||
**Recovery**: Retry with exponential backoff
|
||||
**Fallback**: If retries exhausted, store to local backup, retry next hour
|
||||
**User Impact**: Export shows "delayed", user can check status later
|
||||
|
||||
### Error 4: File Too Large
|
||||
**Trigger**: Export file exceeds 500MB limit
|
||||
**Response Code**: 413 Payload Too Large
|
||||
**Message**: "Export data exceeds 500MB. Use date filtering to reduce size."
|
||||
**Recovery**: User modifies date range and resubmits
|
||||
|
||||
### Timeout Handling
|
||||
**Job Timeout**: If export takes > 5 minutes, job is killed
|
||||
**User Notification**: "Export processing took too long. Please try again."
|
||||
**Logs**: Timeout recorded for analysis
|
||||
**Recovery**: User can request again (usually succeeds second time)
|
||||
```
|
||||
|
||||
### Async/Event Section
|
||||
|
||||
Document asynchronous aspects:
|
||||
|
||||
```markdown
|
||||
## Asynchronous Operations
|
||||
|
||||
### Event: Export Created
|
||||
**Trigger**: POST /exports returns 201
|
||||
**Event Published**: `export.created`
|
||||
**Subscribers**: Analytics service (tracks export requests)
|
||||
**Payload**:
|
||||
```json
|
||||
{
|
||||
"export_id": "exp_123456",
|
||||
"user_id": "usr_789012",
|
||||
"format": "csv",
|
||||
"timestamp": "2024-01-15T10:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Event: Export Completed
|
||||
**Trigger**: Worker successfully uploads to S3
|
||||
**Event Published**: `export.completed`
|
||||
**Subscribers**:
|
||||
- Notification service (send email)
|
||||
- Analytics service (track completion)
|
||||
**Payload**:
|
||||
```json
|
||||
{
|
||||
"export_id": "exp_123456",
|
||||
"file_size_bytes": 2048576,
|
||||
"processing_time_ms": 312000,
|
||||
"timestamp": "2024-01-15T10:05:12Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Event: Export Failed
|
||||
**Trigger**: Job fails after max retries
|
||||
**Event Published**: `export.failed`
|
||||
**Subscribers**: Notification service (alert user)
|
||||
**Payload**:
|
||||
```json
|
||||
{
|
||||
"export_id": "exp_123456",
|
||||
"error_code": "database_timeout",
|
||||
"error_message": "Connection timeout after 3 retries",
|
||||
"timestamp": "2024-01-15T10:06:00Z"
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
### Performance & Timing Section
|
||||
|
||||
```markdown
|
||||
## Performance Characteristics
|
||||
|
||||
### Typical Timings
|
||||
- Request submission → queued: < 100ms
|
||||
- Queued → processing starts: < 30 seconds (depends on queue load)
|
||||
- Processing time:
|
||||
- Small dataset (< 10MB): 1-2 minutes
|
||||
- Medium dataset (10-100MB): 2-5 minutes
|
||||
- Large dataset (100-500MB): 5-10 minutes
|
||||
- Upload to S3: 30 seconds to 2 minutes
|
||||
|
||||
### Total End-to-End Time
|
||||
- Average: 5-10 minutes from request to download ready
|
||||
- Best case: 3-5 minutes (empty queue, small dataset)
|
||||
- Worst case: 15+ minutes (high load, large dataset)
|
||||
|
||||
### Scaling Behavior
|
||||
- 1 worker: Processes 1 export at a time
|
||||
- 3 workers: Process 3 exports in parallel
|
||||
- 10 workers: Can handle 10 concurrent exports
|
||||
- Queue depth auto-scales workers up to 20 pods
|
||||
```
|
||||
|
||||
## Writing Tips
|
||||
|
||||
### Use Clear Diagrams
|
||||
- ASCII art is fine and versioning-friendly
|
||||
- Show all actors and their interactions
|
||||
- Label arrows with what's being transmitted
|
||||
- Use swimlanes for multiple actors
|
||||
|
||||
### Be Specific About Data
|
||||
- Show actual request/response formats
|
||||
- Include field names and types
|
||||
- Show error responses with codes
|
||||
- Document data transformations
|
||||
|
||||
### Cover the Happy Path AND Error Paths
|
||||
- What happens when everything works?
|
||||
- What happens when things go wrong?
|
||||
- What are the recovery mechanisms?
|
||||
- Can users recover?
|
||||
|
||||
### Think About Timing
|
||||
- What happens asynchronously?
|
||||
- Where are synchronous waits?
|
||||
- What are typical timings?
|
||||
- Where are bottlenecks?
|
||||
|
||||
### Link to Related Specs
|
||||
- Reference design documents: `[DES-001]`
|
||||
- Reference API contracts: `[API-001]`
|
||||
- Reference component specs: `[CMP-001]`
|
||||
- Reference data models: `[DATA-001]`
|
||||
|
||||
## Validation & Fixing Issues
|
||||
|
||||
### Run the Validator
|
||||
```bash
|
||||
scripts/validate-spec.sh docs/specs/flow-schematic/flow-001-your-spec.md
|
||||
```
|
||||
|
||||
### Common Issues & Fixes
|
||||
|
||||
**Issue**: "Flow diagram incomplete or missing"
|
||||
- **Fix**: Add ASCII diagram or swimlane showing all steps
|
||||
|
||||
**Issue**: "Step descriptions lack detail"
|
||||
- **Fix**: Add what happens, who's involved, input/output for each step
|
||||
|
||||
**Issue**: "No error handling documented"
|
||||
- **Fix**: Document error cases and recovery mechanisms
|
||||
|
||||
**Issue**: "Async operations not clearly shown"
|
||||
- **Fix**: Highlight asynchronous steps and show event flows
|
||||
|
||||
## Decision-Making Framework
|
||||
|
||||
When documenting a flow:
|
||||
|
||||
1. **Scope**: What does this flow cover?
|
||||
- Where does it start/end?
|
||||
- What's in scope vs. out?
|
||||
|
||||
2. **Actors**: Who/what are the main actors?
|
||||
- Users, systems, services?
|
||||
- External dependencies?
|
||||
|
||||
3. **Happy Path**: What's the ideal flow?
|
||||
- Step-by-step happy path
|
||||
- Minimal branching
|
||||
|
||||
4. **Edge Cases**: What can go wrong?
|
||||
- Error scenarios
|
||||
- Recovery mechanisms
|
||||
- User impact
|
||||
|
||||
5. **Timing**: What's the performance profile?
|
||||
- Synchronous waits?
|
||||
- Asynchronous operations?
|
||||
- Expected timings?
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Create the spec**: `scripts/generate-spec.sh flow-schematic flow-XXX-slug`
|
||||
2. **Research**: Find related specs and understand context
|
||||
3. **Sketch diagram**: Draw initial flow with all actors
|
||||
4. **Document steps**: Write detailed description for each step
|
||||
5. **Add error handling**: Document failure scenarios
|
||||
6. **Validate**: `scripts/validate-spec.sh docs/specs/flow-schematic/flow-XXX-slug.md`
|
||||
7. **Get feedback** from team to refine flow
|
||||
Reference in New Issue
Block a user