Initial commit
This commit is contained in:
Zhongwei Li
82
skills/architecture-diagram-creator/SKILL.md
Normal file
82
skills/architecture-diagram-creator/SKILL.md
Normal file
@@ -0,0 +1,82 @@
|
||||
---
|
||||
name: architecture-diagram-creator
|
||||
description: Create comprehensive HTML architecture diagrams showing data flows, business objectives, features, technical architecture, and deployment. Use when users request system architecture, project documentation, high-level overviews, or technical specifications.
|
||||
---
|
||||
|
||||
# Architecture Diagram Creator
|
||||
|
||||
Create comprehensive HTML architecture diagrams with data flows, business context, and system architecture.
|
||||
|
||||
## When to Use
|
||||
|
||||
- "Create architecture diagram for [project]"
|
||||
- "Generate high-level overview"
|
||||
- "Document system architecture"
|
||||
- "Show data flow and processing pipeline"
|
||||
|
||||
## Components to Include
|
||||
|
||||
1. **Business Context**: objectives, users, value, metrics
|
||||
2. **Data Flow**: sources → processing → outputs with SVG diagram
|
||||
3. **Processing Pipeline**: multi-stage visualization
|
||||
4. **System Architecture**: layered components (data/processing/services/output)
|
||||
5. **Features**: functional and non-functional requirements
|
||||
6. **Deployment**: model, prerequisites, workflows
|
||||
|
||||
## HTML Structure
|
||||
|
||||
```html
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>[Project] Architecture</title>
|
||||
<style>
|
||||
body { font-family: system-ui; max-width: 1200px; margin: 0 auto; padding: 20px; }
|
||||
h1 { background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); color: white; padding: 30px; }
|
||||
.section { margin: 30px 0; }
|
||||
svg { max-width: 100%; }
|
||||
/* Use semantic colors: #4299e1 (data), #ed8936 (processing), #9f7aea (AI), #48bb78 (success) */
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>[Project Name] - Architecture Overview</h1>
|
||||
|
||||
<!-- Business Context Section -->
|
||||
<!-- Data Flow Diagram (SVG) -->
|
||||
<!-- Processing Pipeline (SVG) -->
|
||||
<!-- System Architecture Layers -->
|
||||
<!-- Features Grid -->
|
||||
<!-- Deployment Info -->
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
## SVG Pattern for Data Flow
|
||||
|
||||
```html
|
||||
<svg viewBox="0 0 800 400">
|
||||
<!-- Data sources (left, blue) -->
|
||||
<rect x="50" y="150" width="120" height="80" fill="#4299e1"/>
|
||||
|
||||
<!-- Processing (center, orange) -->
|
||||
<rect x="340" y="150" width="120" height="80" fill="#ed8936"/>
|
||||
|
||||
<!-- Outputs (right, green) -->
|
||||
<rect x="630" y="150" width="120" height="80" fill="#48bb78"/>
|
||||
|
||||
<!-- Arrows connecting -->
|
||||
<path d="M170,190 L340,190" stroke="#666" stroke-width="2" marker-end="url(#arrow)"/>
|
||||
</svg>
|
||||
```
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Analyze project (README, code structure)
|
||||
2. Extract: purpose, data sources, processing, tech stack, outputs
|
||||
3. Create HTML with all 6 sections
|
||||
4. Use semantic colors for visual hierarchy
|
||||
5. Write to `[project]-architecture.html`
|
||||
|
||||
Keep diagrams clear, use consistent styling, include real project details.
|
||||
@@ -0,0 +1,304 @@
|
||||
<!-- ========================================
|
||||
ARCHITECTURE DIAGRAM SVG COMPONENTS
|
||||
Reusable SVG patterns for architecture diagrams
|
||||
======================================== -->
|
||||
|
||||
<!-- ===== SVG MARKERS & DEFINITIONS ===== -->
|
||||
<defs>
|
||||
<!-- Standard arrow marker -->
|
||||
<marker id="arrowhead" markerWidth="10" markerHeight="10"
|
||||
refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#2d3748"/>
|
||||
</marker>
|
||||
|
||||
<!-- Green arrow marker (for success paths) -->
|
||||
<marker id="arrowhead-green" markerWidth="10" markerHeight="10"
|
||||
refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#48bb78"/>
|
||||
</marker>
|
||||
|
||||
<!-- Blue arrow marker (for data flow) -->
|
||||
<marker id="arrowhead-blue" markerWidth="10" markerHeight="10"
|
||||
refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#4299e1"/>
|
||||
</marker>
|
||||
|
||||
<!-- Orange arrow marker (for processing) -->
|
||||
<marker id="arrowhead-orange" markerWidth="10" markerHeight="10"
|
||||
refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#ed8936"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
|
||||
<!-- ===== DATA SOURCE BOX ===== -->
|
||||
<!-- Blue box for data input sources -->
|
||||
<g class="data-source-component">
|
||||
<rect x="50" y="50" width="250" height="200" rx="15"
|
||||
fill="#4299e1" stroke="#2b6cb0" stroke-width="3"/>
|
||||
<text x="175" y="85" text-anchor="middle" fill="white"
|
||||
font-size="16" font-weight="bold">Source Name</text>
|
||||
<text x="175" y="110" text-anchor="middle" fill="white"
|
||||
font-size="12">Source Type</text>
|
||||
|
||||
<!-- Details section -->
|
||||
<text x="175" y="140" text-anchor="middle" fill="white"
|
||||
font-size="11" font-weight="bold">Details:</text>
|
||||
<text x="175" y="160" text-anchor="middle" fill="white"
|
||||
font-size="10">• Field 1</text>
|
||||
<text x="175" y="175" text-anchor="middle" fill="white"
|
||||
font-size="10">• Field 2</text>
|
||||
<text x="175" y="190" text-anchor="middle" fill="white"
|
||||
font-size="10">• Field 3</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== PROCESSING ENGINE BOX ===== -->
|
||||
<!-- Orange box for processing/transformation logic -->
|
||||
<g class="processing-component">
|
||||
<rect x="450" y="200" width="300" height="150" rx="15"
|
||||
fill="#ed8936" stroke="#c05621" stroke-width="3"/>
|
||||
<text x="600" y="235" text-anchor="middle" fill="white"
|
||||
font-size="16" font-weight="bold">Processing Engine</text>
|
||||
|
||||
<!-- Processing steps -->
|
||||
<text x="600" y="260" text-anchor="middle" fill="white"
|
||||
font-size="11">• Step 1: Description</text>
|
||||
<text x="600" y="280" text-anchor="middle" fill="white"
|
||||
font-size="11">• Step 2: Description</text>
|
||||
<text x="600" y="300" text-anchor="middle" fill="white"
|
||||
font-size="11">• Step 3: Description</text>
|
||||
<text x="600" y="320" text-anchor="middle" fill="white"
|
||||
font-size="11">• Step 4: Description</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== OUTPUT/RESULT BOX ===== -->
|
||||
<!-- Green box for outputs and results -->
|
||||
<g class="output-component">
|
||||
<rect x="900" y="175" width="250" height="200" rx="15"
|
||||
fill="#48bb78" stroke="#2f855a" stroke-width="3"/>
|
||||
<text x="1025" y="210" text-anchor="middle" fill="white"
|
||||
font-size="16" font-weight="bold">Output Name</text>
|
||||
<text x="1025" y="235" text-anchor="middle" fill="white"
|
||||
font-size="11" font-weight="bold">Output Details:</text>
|
||||
|
||||
<!-- Output details -->
|
||||
<text x="1025" y="255" text-anchor="middle" fill="white"
|
||||
font-size="10">• Format: Type</text>
|
||||
<text x="1025" y="270" text-anchor="middle" fill="white"
|
||||
font-size="10">• Size: Amount</text>
|
||||
<text x="1025" y="285" text-anchor="middle" fill="white"
|
||||
font-size="10">• Destination: Location</text>
|
||||
|
||||
<!-- Status indicators -->
|
||||
<text x="1025" y="310" text-anchor="middle" fill="white"
|
||||
font-size="10">✓ Status indicator</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== AI/ML SERVICE BOX ===== -->
|
||||
<!-- Purple box for AI/ML services -->
|
||||
<g class="ai-service-component">
|
||||
<rect x="320" y="400" width="250" height="120" rx="15"
|
||||
fill="#9f7aea" stroke="#6b46c1" stroke-width="3"/>
|
||||
<text x="445" y="435" text-anchor="middle" fill="white"
|
||||
font-size="16" font-weight="bold">AI Service</text>
|
||||
<text x="445" y="460" text-anchor="middle" fill="white"
|
||||
font-size="11">Model/Provider Name</text>
|
||||
<text x="445" y="480" text-anchor="middle" fill="white"
|
||||
font-size="10">• Capability 1</text>
|
||||
<text x="445" y="495" text-anchor="middle" fill="white"
|
||||
font-size="10">• Capability 2</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== CONFIGURATION BOX ===== -->
|
||||
<!-- Amber box for configuration and settings -->
|
||||
<g class="config-component">
|
||||
<rect x="470" y="60" width="180" height="100" rx="10"
|
||||
fill="#f59e0b" stroke="#d97706" stroke-width="2"/>
|
||||
<text x="560" y="90" text-anchor="middle" fill="white"
|
||||
font-size="14" font-weight="bold">Configuration</text>
|
||||
<text x="560" y="110" text-anchor="middle" fill="white"
|
||||
font-size="10">Config file</text>
|
||||
<text x="560" y="125" text-anchor="middle" fill="white"
|
||||
font-size="10">• Settings</text>
|
||||
<text x="560" y="140" text-anchor="middle" fill="white"
|
||||
font-size="10">• Parameters</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== SUPPORTING TOOL BOX ===== -->
|
||||
<!-- Gray box for supporting tools/utilities -->
|
||||
<g class="tool-component">
|
||||
<rect x="750" y="240" width="200" height="70" rx="10"
|
||||
fill="#718096" stroke="#4a5568" stroke-width="2"/>
|
||||
<text x="850" y="265" text-anchor="middle" fill="white"
|
||||
font-size="11" font-weight="bold">Tool Name</text>
|
||||
<text x="850" y="283" text-anchor="middle" fill="white"
|
||||
font-size="9">Description</text>
|
||||
<text x="850" y="297" text-anchor="middle" fill="white"
|
||||
font-size="9">Purpose</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== PIPELINE STAGE BOX ===== -->
|
||||
<!-- Compact pipeline stage for multi-stage diagrams -->
|
||||
<g class="pipeline-stage">
|
||||
<rect x="50" y="50" width="200" height="100" rx="10"
|
||||
fill="#4299e1" stroke="#2b6cb0" stroke-width="2"/>
|
||||
<text x="150" y="80" text-anchor="middle" fill="white"
|
||||
font-size="14" font-weight="bold">Stage Name</text>
|
||||
<text x="150" y="100" text-anchor="middle" fill="white"
|
||||
font-size="10">Operation</text>
|
||||
<text x="150" y="115" text-anchor="middle" fill="white"
|
||||
font-size="10">Technology</text>
|
||||
<text x="150" y="130" text-anchor="middle" fill="white"
|
||||
font-size="10">Output format</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== LAYERED ARCHITECTURE COMPONENT ===== -->
|
||||
<!-- Stack of layers for architecture diagrams -->
|
||||
<g class="architecture-layer">
|
||||
<!-- Layer 1 -->
|
||||
<rect x="100" y="100" width="400" height="80" rx="10"
|
||||
fill="#4299e1" stroke="#2b6cb0" stroke-width="2"/>
|
||||
<text x="300" y="145" text-anchor="middle" fill="white"
|
||||
font-size="13" font-weight="bold">Layer 1: Data Sources</text>
|
||||
|
||||
<!-- Layer 2 -->
|
||||
<rect x="100" y="200" width="400" height="80" rx="10"
|
||||
fill="#ed8936" stroke="#c05621" stroke-width="2"/>
|
||||
<text x="300" y="245" text-anchor="middle" fill="white"
|
||||
font-size="13" font-weight="bold">Layer 2: Processing</text>
|
||||
|
||||
<!-- Layer 3 -->
|
||||
<rect x="100" y="300" width="400" height="80" rx="10"
|
||||
fill="#9f7aea" stroke="#6b46c1" stroke-width="2"/>
|
||||
<text x="300" y="345" text-anchor="middle" fill="white"
|
||||
font-size="13" font-weight="bold">Layer 3: Services</text>
|
||||
|
||||
<!-- Layer 4 -->
|
||||
<rect x="100" y="400" width="400" height="80" rx="10"
|
||||
fill="#48bb78" stroke="#2f855a" stroke-width="2"/>
|
||||
<text x="300" y="445" text-anchor="middle" fill="white"
|
||||
font-size="13" font-weight="bold">Layer 4: Output</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== CONNECTORS & ARROWS ===== -->
|
||||
<!-- Simple horizontal arrow -->
|
||||
<g class="arrow-horizontal">
|
||||
<path d="M 300 150 L 450 150" stroke="#2d3748"
|
||||
stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
<text x="375" y="140" text-anchor="middle" fill="#2d3748"
|
||||
font-size="11" font-weight="bold">Label</text>
|
||||
</g>
|
||||
|
||||
<!-- Diagonal arrow (down-right) -->
|
||||
<g class="arrow-diagonal">
|
||||
<path d="M 300 150 L 450 250" stroke="#2d3748"
|
||||
stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
<text x="360" y="190" fill="#2d3748"
|
||||
font-size="11" font-weight="bold">Label</text>
|
||||
</g>
|
||||
|
||||
<!-- Vertical arrow (downward) -->
|
||||
<g class="arrow-vertical">
|
||||
<path d="M 150 150 L 150 280" stroke="#2d3748"
|
||||
stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
<text x="160" y="215" fill="#2d3748"
|
||||
font-size="11" font-weight="bold">Label</text>
|
||||
</g>
|
||||
|
||||
<!-- L-shaped connector (right then down) -->
|
||||
<g class="connector-l-shape">
|
||||
<path d="M 300 150 L 450 150 L 450 280" stroke="#2d3748"
|
||||
stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
</g>
|
||||
|
||||
<!-- Dashed line (for optional/uncertain paths) -->
|
||||
<g class="arrow-dashed">
|
||||
<path d="M 300 150 L 450 150" stroke="#718096"
|
||||
stroke-width="2" stroke-dasharray="5,5"
|
||||
marker-end="url(#arrowhead)"/>
|
||||
<text x="375" y="140" text-anchor="middle" fill="#718096"
|
||||
font-size="10" font-style="italic">Optional</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== ANNOTATION/CALLOUT BOX ===== -->
|
||||
<!-- Info box with border and background -->
|
||||
<g class="annotation-box">
|
||||
<rect x="880" y="150" width="220" height="120" rx="10"
|
||||
fill="#fff5f5" stroke="#fc8181" stroke-width="2"/>
|
||||
<text x="990" y="180" text-anchor="middle" fill="#742a2a"
|
||||
font-size="12" font-weight="bold">⚠️ Important Note</text>
|
||||
<text x="990" y="200" text-anchor="middle" fill="#5a1919"
|
||||
font-size="9">Line 1 of note text</text>
|
||||
<text x="990" y="215" text-anchor="middle" fill="#5a1919"
|
||||
font-size="9">Line 2 of note text</text>
|
||||
<text x="990" y="230" text-anchor="middle" fill="#5a1919"
|
||||
font-size="9">Line 3 of note text</text>
|
||||
<text x="990" y="245" text-anchor="middle" fill="#5a1919"
|
||||
font-size="9">Line 4 of note text</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== REFERENCE TABLE ROW ===== -->
|
||||
<!-- Table header row -->
|
||||
<g class="table-header">
|
||||
<rect x="50" y="50" width="140" height="40"
|
||||
fill="#667eea" stroke="#5a67d8" stroke-width="2"/>
|
||||
<text x="120" y="75" text-anchor="middle" fill="white"
|
||||
font-size="12" font-weight="bold">Column 1</text>
|
||||
|
||||
<rect x="190" y="50" width="180" height="40"
|
||||
fill="#667eea" stroke="#5a67d8" stroke-width="2"/>
|
||||
<text x="280" y="75" text-anchor="middle" fill="white"
|
||||
font-size="12" font-weight="bold">Column 2</text>
|
||||
|
||||
<rect x="370" y="50" width="140" height="40"
|
||||
fill="#667eea" stroke="#5a67d8" stroke-width="2"/>
|
||||
<text x="440" y="75" text-anchor="middle" fill="white"
|
||||
font-size="12" font-weight="bold">Column 3</text>
|
||||
</g>
|
||||
|
||||
<!-- Table data row -->
|
||||
<g class="table-row">
|
||||
<rect x="50" y="90" width="140" height="50"
|
||||
fill="#e6fffa" stroke="#81e6d9" stroke-width="1"/>
|
||||
<text x="120" y="120" text-anchor="middle" fill="#234e52"
|
||||
font-size="11" font-weight="bold">Data 1</text>
|
||||
|
||||
<rect x="190" y="90" width="180" height="50"
|
||||
fill="#e6fffa" stroke="#81e6d9" stroke-width="1"/>
|
||||
<text x="280" y="120" text-anchor="middle" fill="#234e52"
|
||||
font-size="10">Data 2</text>
|
||||
|
||||
<rect x="370" y="90" width="140" height="50"
|
||||
fill="#e6fffa" stroke="#81e6d9" stroke-width="1"/>
|
||||
<text x="440" y="120" text-anchor="middle" fill="#234e52"
|
||||
font-size="10">Data 3</text>
|
||||
</g>
|
||||
|
||||
|
||||
<!-- ===== COLOR PALETTE REFERENCE ===== -->
|
||||
<!--
|
||||
Data Sources / Inputs: #4299e1 (blue) stroke: #2b6cb0
|
||||
Processing / Logic: #ed8936 (orange) stroke: #c05621
|
||||
AI/ML Services: #9f7aea (purple) stroke: #6b46c1
|
||||
Output / Success: #48bb78 (green) stroke: #2f855a
|
||||
Configuration: #f59e0b (amber) stroke: #d97706
|
||||
Supporting Tools: #718096 (gray) stroke: #4a5568
|
||||
Warning / Critical: #e53e3e (red) stroke: #c53030
|
||||
Information / Teal: #38b2ac (teal) stroke: #2c7a7b
|
||||
Special / Highlight: #805ad5 (purple2) stroke: #6b46c1
|
||||
Text dark: #2d3748
|
||||
Text medium: #4a5568
|
||||
Text light: #718096
|
||||
Background light: #f7fafc
|
||||
Border light: #e2e8f0
|
||||
-->
|
||||
@@ -0,0 +1,306 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>[PROJECT_NAME] Architecture</title>
|
||||
<style>
|
||||
/* ===== RESET & BASE ===== */
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
padding: 2rem;
|
||||
min-height: 100vh;
|
||||
}
|
||||
|
||||
/* ===== CONTAINER ===== */
|
||||
.container {
|
||||
max-width: 1400px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
border-radius: 20px;
|
||||
box-shadow: 0 20px 60px rgba(0,0,0,0.3);
|
||||
padding: 3rem;
|
||||
}
|
||||
|
||||
/* ===== TYPOGRAPHY ===== */
|
||||
h1 {
|
||||
font-size: 2.5rem;
|
||||
color: #2d3748;
|
||||
margin-bottom: 0.5rem;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
text-align: center;
|
||||
color: #718096;
|
||||
font-size: 1.1rem;
|
||||
margin-bottom: 2rem;
|
||||
}
|
||||
|
||||
/* ===== SECTIONS ===== */
|
||||
.section {
|
||||
margin: 2rem 0;
|
||||
}
|
||||
|
||||
.section-title {
|
||||
font-size: 1.5rem;
|
||||
color: #2d3748;
|
||||
margin-bottom: 1rem;
|
||||
padding-bottom: 0.5rem;
|
||||
border-bottom: 3px solid #667eea;
|
||||
}
|
||||
|
||||
/* ===== METRIC CARDS ===== */
|
||||
.metric-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||
gap: 1rem;
|
||||
margin-bottom: 2rem;
|
||||
}
|
||||
|
||||
.metric-card {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
padding: 1.5rem;
|
||||
border-radius: 12px;
|
||||
text-align: center;
|
||||
color: white;
|
||||
}
|
||||
|
||||
.metric-value {
|
||||
font-size: 2rem;
|
||||
font-weight: bold;
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.metric-label {
|
||||
font-size: 0.9rem;
|
||||
opacity: 0.9;
|
||||
}
|
||||
|
||||
/* ===== FEATURE CARDS ===== */
|
||||
.feature-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
|
||||
gap: 1.5rem;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
|
||||
.feature-card {
|
||||
background: white;
|
||||
border: 2px solid #e2e8f0;
|
||||
border-radius: 12px;
|
||||
padding: 1.5rem;
|
||||
transition: transform 0.2s, box-shadow 0.2s;
|
||||
}
|
||||
|
||||
.feature-card:hover {
|
||||
transform: translateY(-4px);
|
||||
box-shadow: 0 10px 30px rgba(0,0,0,0.1);
|
||||
}
|
||||
|
||||
.feature-card h3 {
|
||||
color: #667eea;
|
||||
margin-bottom: 0.5rem;
|
||||
font-size: 1.2rem;
|
||||
}
|
||||
|
||||
.feature-card ul {
|
||||
list-style: none;
|
||||
color: #4a5568;
|
||||
line-height: 1.8;
|
||||
}
|
||||
|
||||
.feature-card ul li:before {
|
||||
content: "→ ";
|
||||
color: #667eea;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
/* ===== DIAGRAMS ===== */
|
||||
.diagram-container {
|
||||
background: #f7fafc;
|
||||
border-radius: 12px;
|
||||
padding: 2rem;
|
||||
margin: 1rem 0;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
svg {
|
||||
width: 100%;
|
||||
height: auto;
|
||||
display: block;
|
||||
}
|
||||
|
||||
/* ===== LEGEND ===== */
|
||||
.legend {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 1.5rem;
|
||||
margin-top: 2rem;
|
||||
padding: 1.5rem;
|
||||
background: #f7fafc;
|
||||
border-radius: 12px;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.legend-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.legend-box {
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
border-radius: 4px;
|
||||
border: 2px solid #2d3748;
|
||||
}
|
||||
|
||||
/* ===== FOOTER ===== */
|
||||
footer {
|
||||
margin-top: 3rem;
|
||||
padding-top: 2rem;
|
||||
border-top: 2px solid #e2e8f0;
|
||||
text-align: center;
|
||||
color: #718096;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
|
||||
/* ===== RESPONSIVE ===== */
|
||||
@media (max-width: 768px) {
|
||||
body {
|
||||
padding: 1rem;
|
||||
}
|
||||
|
||||
.container {
|
||||
padding: 1.5rem;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 1.8rem;
|
||||
}
|
||||
|
||||
.metric-grid,
|
||||
.feature-grid {
|
||||
grid-template-columns: 1fr;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<!-- HEADER -->
|
||||
<h1>[PROJECT_ICON] [PROJECT_NAME]</h1>
|
||||
<p class="subtitle">[PROJECT_DESCRIPTION]</p>
|
||||
|
||||
<!-- KEY METRICS -->
|
||||
<div class="metric-grid">
|
||||
<!-- Metric cards will be inserted here -->
|
||||
</div>
|
||||
|
||||
<!-- BUSINESS OBJECTIVES -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">📊 Business Objectives & End Users</h2>
|
||||
<div class="feature-grid">
|
||||
<!-- Feature cards will be inserted here -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- DATA INPUT OVERVIEW -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">📥 Data Input Overview</h2>
|
||||
<div class="diagram-container">
|
||||
<svg viewBox="0 0 1200 600">
|
||||
<!-- Data flow diagram will be inserted here -->
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- PROCESSING PIPELINE -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">⚙️ Data Processing Pipeline</h2>
|
||||
<div class="diagram-container">
|
||||
<svg viewBox="0 0 1400 700">
|
||||
<!-- Pipeline diagram will be inserted here -->
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- FUNCTIONAL FEATURES -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">✨ Functional Features</h2>
|
||||
<div class="feature-grid">
|
||||
<!-- Feature cards will be inserted here -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- NON-FUNCTIONAL FEATURES -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">🛡️ Non-Functional Features</h2>
|
||||
<div class="feature-grid">
|
||||
<!-- Feature cards will be inserted here -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- SYSTEM ARCHITECTURE -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">🏗️ System Architecture</h2>
|
||||
<div class="diagram-container">
|
||||
<svg viewBox="0 0 1400 800">
|
||||
<!-- Architecture diagram will be inserted here -->
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- DEPLOYMENT & USAGE -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">🚀 Deployment & Usage</h2>
|
||||
<div class="feature-grid">
|
||||
<!-- Deployment info cards will be inserted here -->
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- LEGEND -->
|
||||
<div class="legend">
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #4299e1;"></div>
|
||||
<span>Data Sources</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #ed8936;"></div>
|
||||
<span>Processing Logic</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #9f7aea;"></div>
|
||||
<span>AI Services</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #48bb78;"></div>
|
||||
<span>Output/Results</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #f59e0b;"></div>
|
||||
<span>Configuration</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #718096;"></div>
|
||||
<span>Supporting Tools</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- FOOTER -->
|
||||
<footer>
|
||||
<strong>[PROJECT_NAME] Architecture v[VERSION]</strong><br>
|
||||
Generated: [DATE] | [SHORT_DESCRIPTION]<br>
|
||||
Technologies: [TECH_STACK]
|
||||
</footer>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,603 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>DataFlow ETL Pipeline Architecture</title>
|
||||
<style>
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
padding: 2rem;
|
||||
min-height: 100vh;
|
||||
}
|
||||
|
||||
.container {
|
||||
max-width: 1400px;
|
||||
margin: 0 auto;
|
||||
background: white;
|
||||
border-radius: 20px;
|
||||
box-shadow: 0 20px 60px rgba(0,0,0,0.3);
|
||||
padding: 3rem;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 2.5rem;
|
||||
color: #2d3748;
|
||||
margin-bottom: 0.5rem;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
text-align: center;
|
||||
color: #718096;
|
||||
font-size: 1.1rem;
|
||||
margin-bottom: 2rem;
|
||||
}
|
||||
|
||||
.metric-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
|
||||
gap: 1rem;
|
||||
margin-bottom: 2rem;
|
||||
}
|
||||
|
||||
.metric-card {
|
||||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||
padding: 1.5rem;
|
||||
border-radius: 12px;
|
||||
text-align: center;
|
||||
color: white;
|
||||
}
|
||||
|
||||
.metric-value {
|
||||
font-size: 2rem;
|
||||
font-weight: bold;
|
||||
margin-bottom: 0.5rem;
|
||||
}
|
||||
|
||||
.metric-label {
|
||||
font-size: 0.9rem;
|
||||
opacity: 0.9;
|
||||
}
|
||||
|
||||
.section {
|
||||
margin: 2rem 0;
|
||||
}
|
||||
|
||||
.section-title {
|
||||
font-size: 1.5rem;
|
||||
color: #2d3748;
|
||||
margin-bottom: 1rem;
|
||||
padding-bottom: 0.5rem;
|
||||
border-bottom: 3px solid #667eea;
|
||||
}
|
||||
|
||||
.diagram-container {
|
||||
background: #f7fafc;
|
||||
border-radius: 12px;
|
||||
padding: 2rem;
|
||||
margin: 1rem 0;
|
||||
overflow-x: auto;
|
||||
}
|
||||
|
||||
svg {
|
||||
width: 100%;
|
||||
height: auto;
|
||||
display: block;
|
||||
}
|
||||
|
||||
.legend {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 1.5rem;
|
||||
margin-top: 2rem;
|
||||
padding: 1.5rem;
|
||||
background: #f7fafc;
|
||||
border-radius: 12px;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.legend-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.legend-box {
|
||||
width: 20px;
|
||||
height: 20px;
|
||||
border-radius: 4px;
|
||||
border: 2px solid #2d3748;
|
||||
}
|
||||
|
||||
.feature-grid {
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
|
||||
gap: 1.5rem;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
|
||||
.feature-card {
|
||||
background: white;
|
||||
border: 2px solid #e2e8f0;
|
||||
border-radius: 12px;
|
||||
padding: 1.5rem;
|
||||
transition: transform 0.2s, box-shadow 0.2s;
|
||||
}
|
||||
|
||||
.feature-card:hover {
|
||||
transform: translateY(-4px);
|
||||
box-shadow: 0 10px 30px rgba(0,0,0,0.1);
|
||||
}
|
||||
|
||||
.feature-card h3 {
|
||||
color: #667eea;
|
||||
margin-bottom: 0.5rem;
|
||||
font-size: 1.2rem;
|
||||
}
|
||||
|
||||
.feature-card ul {
|
||||
list-style: none;
|
||||
color: #4a5568;
|
||||
line-height: 1.8;
|
||||
}
|
||||
|
||||
.feature-card ul li:before {
|
||||
content: "→ ";
|
||||
color: #667eea;
|
||||
font-weight: bold;
|
||||
}
|
||||
|
||||
footer {
|
||||
margin-top: 3rem;
|
||||
padding-top: 2rem;
|
||||
border-top: 2px solid #e2e8f0;
|
||||
text-align: center;
|
||||
color: #718096;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="container">
|
||||
<h1>🔄 DataFlow ETL Pipeline</h1>
|
||||
<p class="subtitle">Customer Data Integration & Analytics Platform</p>
|
||||
|
||||
<!-- Key Metrics -->
|
||||
<div class="metric-grid">
|
||||
<div class="metric-card">
|
||||
<div class="metric-value">3</div>
|
||||
<div class="metric-label">Data Sources</div>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<div class="metric-value">5</div>
|
||||
<div class="metric-label">Processing Stages</div>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<div class="metric-value">100K</div>
|
||||
<div class="metric-label">Records/Day</div>
|
||||
</div>
|
||||
<div class="metric-card">
|
||||
<div class="metric-value">99.9%</div>
|
||||
<div class="metric-label">Uptime SLA</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Business Context -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">📊 Business Objectives & End Users</h2>
|
||||
<div class="feature-grid">
|
||||
<div class="feature-card">
|
||||
<h3>Primary Objective</h3>
|
||||
<ul>
|
||||
<li>Consolidate customer data from multiple sources</li>
|
||||
<li>Provide unified view for analytics and reporting</li>
|
||||
<li>Enable real-time data-driven decision making</li>
|
||||
<li>Ensure data quality and consistency</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>End Users</h3>
|
||||
<ul>
|
||||
<li>Business Analysts (data exploration)</li>
|
||||
<li>Data Scientists (ML model training)</li>
|
||||
<li>Marketing Team (campaign targeting)</li>
|
||||
<li>Customer Success (account insights)</li>
|
||||
<li>Executive Dashboard (KPI monitoring)</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Business Value</h3>
|
||||
<ul>
|
||||
<li>Reduce manual data reconciliation (80% time savings)</li>
|
||||
<li>Improve data accuracy and completeness</li>
|
||||
<li>Enable faster business insights</li>
|
||||
<li>Scale data processing capacity</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Data Input Overview -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">📥 Data Input Overview</h2>
|
||||
<div class="diagram-container">
|
||||
<svg viewBox="0 0 1200 500">
|
||||
<defs>
|
||||
<marker id="arrowhead" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#2d3748"/>
|
||||
</marker>
|
||||
<marker id="arrowhead-green" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#48bb78"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- Source 1: CRM API -->
|
||||
<rect x="50" y="50" width="220" height="150" rx="15" fill="#4299e1" stroke="#2b6cb0" stroke-width="3"/>
|
||||
<text x="160" y="85" text-anchor="middle" fill="white" font-size="16" font-weight="bold">Source 1: CRM API</text>
|
||||
<text x="160" y="110" text-anchor="middle" fill="white" font-size="12">(Salesforce)</text>
|
||||
<text x="160" y="135" text-anchor="middle" fill="white" font-size="10">Format: JSON REST API</text>
|
||||
<text x="160" y="150" text-anchor="middle" fill="white" font-size="10">~50K records/day</text>
|
||||
<text x="160" y="165" text-anchor="middle" fill="white" font-size="10">Customer profiles</text>
|
||||
<text x="160" y="180" text-anchor="middle" fill="white" font-size="10">Real-time sync</text>
|
||||
|
||||
<!-- Source 2: Database -->
|
||||
<rect x="50" y="220" width="220" height="150" rx="15" fill="#9f7aea" stroke="#6b46c1" stroke-width="3"/>
|
||||
<text x="160" y="255" text-anchor="middle" fill="white" font-size="16" font-weight="bold">Source 2: Orders DB</text>
|
||||
<text x="160" y="280" text-anchor="middle" fill="white" font-size="12">(MySQL)</text>
|
||||
<text x="160" y="305" text-anchor="middle" fill="white" font-size="10">Format: SQL queries</text>
|
||||
<text x="160" y="320" text-anchor="middle" fill="white" font-size="10">~30K orders/day</text>
|
||||
<text x="160" y="335" text-anchor="middle" fill="white" font-size="10">Transaction data</text>
|
||||
<text x="160" y="350" text-anchor="middle" fill="white" font-size="10">Hourly batch</text>
|
||||
|
||||
<!-- Source 3: CSV Files -->
|
||||
<rect x="50" y="390" width="220" height="80" rx="15" fill="#ed8936" stroke="#c05621" stroke-width="3"/>
|
||||
<text x="160" y="420" text-anchor="middle" fill="white" font-size="16" font-weight="bold">Source 3: CSV Export</text>
|
||||
<text x="160" y="440" text-anchor="middle" fill="white" font-size="10">~20K records/day</text>
|
||||
<text x="160" y="455" text-anchor="middle" fill="white" font-size="10">Support tickets (S3)</text>
|
||||
|
||||
<!-- ETL Pipeline -->
|
||||
<rect x="400" y="175" width="280" height="150" rx="15" fill="#ed8936" stroke="#c05621" stroke-width="3"/>
|
||||
<text x="540" y="210" text-anchor="middle" fill="white" font-size="16" font-weight="bold">ETL Pipeline</text>
|
||||
<text x="540" y="235" text-anchor="middle" fill="white" font-size="11">AWS Lambda + Airflow</text>
|
||||
<text x="540" y="260" text-anchor="middle" fill="white" font-size="10">• Data validation</text>
|
||||
<text x="540" y="278" text-anchor="middle" fill="white" font-size="10">• Schema transformation</text>
|
||||
<text x="540" y="296" text-anchor="middle" fill="white" font-size="10">• Deduplication</text>
|
||||
<text x="540" y="314" text-anchor="middle" fill="white" font-size="10">• Enrichment</text>
|
||||
|
||||
<!-- Data Warehouse -->
|
||||
<rect x="820" y="150" width="300" height="200" rx="15" fill="#48bb78" stroke="#2f855a" stroke-width="3"/>
|
||||
<text x="970" y="190" text-anchor="middle" fill="white" font-size="16" font-weight="bold">Data Warehouse</text>
|
||||
<text x="970" y="215" text-anchor="middle" fill="white" font-size="12">(BigQuery)</text>
|
||||
<text x="970" y="245" text-anchor="middle" fill="white" font-size="10">Unified customer view</text>
|
||||
<text x="970" y="263" text-anchor="middle" fill="white" font-size="10">360° analytics</text>
|
||||
<text x="970" y="281" text-anchor="middle" fill="white" font-size="10">Historical trends</text>
|
||||
<text x="970" y="299" text-anchor="middle" fill="white" font-size="10">ML-ready datasets</text>
|
||||
<text x="970" y="325" text-anchor="middle" fill="white" font-size="10">✓ GDPR compliant</text>
|
||||
|
||||
<!-- Arrows -->
|
||||
<path d="M 270 125 L 400 225" stroke="#2d3748" stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
<path d="M 270 295 L 400 275" stroke="#2d3748" stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
<path d="M 270 430 L 400 300" stroke="#2d3748" stroke-width="3" marker-end="url(#arrowhead)"/>
|
||||
<path d="M 680 250 L 820 250" stroke="#48bb78" stroke-width="3" marker-end="url(#arrowhead-green)"/>
|
||||
|
||||
<!-- Labels -->
|
||||
<text x="330" y="165" fill="#2d3748" font-size="10" font-weight="bold">Customer data</text>
|
||||
<text x="330" y="285" fill="#2d3748" font-size="10" font-weight="bold">Order data</text>
|
||||
<text x="330" y="370" fill="#2d3748" font-size="10" font-weight="bold">Support data</text>
|
||||
<text x="740" y="240" fill="#48bb78" font-size="11" font-weight="bold">Processed</text>
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Processing Pipeline -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">⚙️ Data Processing Pipeline</h2>
|
||||
<div class="diagram-container">
|
||||
<svg viewBox="0 0 1400 600">
|
||||
<defs>
|
||||
<marker id="arrow-pipeline" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#2d3748"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- Stage 1 -->
|
||||
<rect x="50" y="50" width="200" height="100" rx="10" fill="#4299e1" stroke="#2b6cb0" stroke-width="2"/>
|
||||
<text x="150" y="80" text-anchor="middle" fill="white" font-size="14" font-weight="bold">1. Data Ingestion</text>
|
||||
<text x="150" y="100" text-anchor="middle" fill="white" font-size="10">Pull from sources</text>
|
||||
<text x="150" y="115" text-anchor="middle" fill="white" font-size="10">API + SQL + S3</text>
|
||||
<text x="150" y="130" text-anchor="middle" fill="white" font-size="10">Raw data storage</text>
|
||||
|
||||
<!-- Stage 2 -->
|
||||
<rect x="50" y="180" width="200" height="100" rx="10" fill="#ed8936" stroke="#c05621" stroke-width="2"/>
|
||||
<text x="150" y="210" text-anchor="middle" fill="white" font-size="14" font-weight="bold">2. Validation</text>
|
||||
<text x="150" y="230" text-anchor="middle" fill="white" font-size="10">Schema checks</text>
|
||||
<text x="150" y="245" text-anchor="middle" fill="white" font-size="10">Data quality rules</text>
|
||||
<text x="150" y="260" text-anchor="middle" fill="white" font-size="10">Error logging</text>
|
||||
|
||||
<!-- Stage 3 -->
|
||||
<rect x="50" y="310" width="200" height="100" rx="10" fill="#9f7aea" stroke="#6b46c1" stroke-width="2"/>
|
||||
<text x="150" y="340" text-anchor="middle" fill="white" font-size="14" font-weight="bold">3. Transformation</text>
|
||||
<text x="150" y="360" text-anchor="middle" fill="white" font-size="10">Normalize formats</text>
|
||||
<text x="150" y="375" text-anchor="middle" fill="white" font-size="10">Map fields</text>
|
||||
<text x="150" y="390" text-anchor="middle" fill="white" font-size="10">Type conversions</text>
|
||||
|
||||
<!-- Stage 4 -->
|
||||
<rect x="320" y="50" width="200" height="120" rx="10" fill="#38b2ac" stroke="#2c7a7b" stroke-width="2"/>
|
||||
<text x="420" y="80" text-anchor="middle" fill="white" font-size="14" font-weight="bold">4. Deduplication</text>
|
||||
<text x="420" y="100" text-anchor="middle" fill="white" font-size="10">Fuzzy matching</text>
|
||||
<text x="420" y="115" text-anchor="middle" fill="white" font-size="10">Customer ID merge</text>
|
||||
<text x="420" y="130" text-anchor="middle" fill="white" font-size="10">Conflict resolution</text>
|
||||
<text x="420" y="145" text-anchor="middle" fill="white" font-size="10">Master record creation</text>
|
||||
|
||||
<!-- Stage 5 -->
|
||||
<rect x="320" y="200" width="200" height="100" rx="10" fill="#805ad5" stroke="#6b46c1" stroke-width="2"/>
|
||||
<text x="420" y="230" text-anchor="middle" fill="white" font-size="14" font-weight="bold">5. Enrichment</text>
|
||||
<text x="420" y="250" text-anchor="middle" fill="white" font-size="10">Geo-location lookup</text>
|
||||
<text x="420" y="265" text-anchor="middle" fill="white" font-size="10">Industry tagging</text>
|
||||
<text x="420" y="280" text-anchor="middle" fill="white" font-size="10">Score calculations</text>
|
||||
|
||||
<!-- Stage 6 -->
|
||||
<rect x="590" y="100" width="200" height="120" rx="10" fill="#48bb78" stroke="#2f855a" stroke-width="2"/>
|
||||
<text x="690" y="130" text-anchor="middle" fill="white" font-size="14" font-weight="bold">6. Load</text>
|
||||
<text x="690" y="150" text-anchor="middle" fill="white" font-size="10">Write to warehouse</text>
|
||||
<text x="690" y="165" text-anchor="middle" fill="white" font-size="10">Update indexes</text>
|
||||
<text x="690" y="180" text-anchor="middle" fill="white" font-size="10">Trigger downstream</text>
|
||||
<text x="690" y="195" text-anchor="middle" fill="white" font-size="10">✓ Complete</text>
|
||||
|
||||
<!-- Arrows -->
|
||||
<path d="M 150 150 L 150 180" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-pipeline)"/>
|
||||
<path d="M 150 280 L 150 310" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-pipeline)"/>
|
||||
<path d="M 250 100 L 320 110" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-pipeline)"/>
|
||||
<path d="M 420 170 L 420 200" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-pipeline)"/>
|
||||
<path d="M 520 250 L 590 180" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-pipeline)"/>
|
||||
|
||||
<!-- Configuration note -->
|
||||
<rect x="900" y="100" width="400" height="180" rx="10" fill="#f7fafc" stroke="#cbd5e0" stroke-width="2"/>
|
||||
<text x="1100" y="135" text-anchor="middle" fill="#2d3748" font-size="13" font-weight="bold">Pipeline Configuration</text>
|
||||
<text x="930" y="165" fill="#4a5568" font-size="10" font-weight="bold">Orchestration:</text>
|
||||
<text x="930" y="180" fill="#4a5568" font-size="9">• Apache Airflow (DAG scheduling)</text>
|
||||
<text x="930" y="195" fill="#4a5568" font-size="9">• AWS Lambda (serverless compute)</text>
|
||||
<text x="930" y="210" fill="#4a5568" font-size="9">• S3 (intermediate storage)</text>
|
||||
<text x="930" y="230" fill="#4a5568" font-size="9" font-weight="bold">Monitoring:</text>
|
||||
<text x="930" y="245" fill="#4a5568" font-size="9">• CloudWatch logs & metrics</text>
|
||||
<text x="930" y="260" fill="#4a5568" font-size="9">• PagerDuty alerts</text>
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Functional Features -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">✨ Functional Features</h2>
|
||||
<div class="feature-grid">
|
||||
<div class="feature-card">
|
||||
<h3>Data Validation</h3>
|
||||
<ul>
|
||||
<li>JSON schema validation for API data</li>
|
||||
<li>SQL constraint checks for database records</li>
|
||||
<li>Custom business rule engine</li>
|
||||
<li>Automated error notifications</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Intelligent Deduplication</h3>
|
||||
<ul>
|
||||
<li>Fuzzy string matching (Levenshtein distance)</li>
|
||||
<li>Multi-field entity resolution</li>
|
||||
<li>Confidence scoring for matches</li>
|
||||
<li>Manual review queue for uncertain cases</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Data Enrichment</h3>
|
||||
<ul>
|
||||
<li>Geo-location from IP/address</li>
|
||||
<li>Company firmographic data</li>
|
||||
<li>Industry classification</li>
|
||||
<li>Customer lifecycle scoring</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Non-Functional Features -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">🛡️ Non-Functional Features</h2>
|
||||
<div class="feature-grid">
|
||||
<div class="feature-card">
|
||||
<h3>Performance</h3>
|
||||
<ul>
|
||||
<li>Processes 100K records in <30 minutes</li>
|
||||
<li>Parallel processing across 10 Lambda workers</li>
|
||||
<li>Optimized SQL queries with indexes</li>
|
||||
<li>Incremental data loading strategy</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Reliability</h3>
|
||||
<ul>
|
||||
<li>99.9% uptime SLA</li>
|
||||
<li>Automatic retry with exponential backoff</li>
|
||||
<li>Dead-letter queue for failed records</li>
|
||||
<li>Point-in-time recovery capability</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Security & Compliance</h3>
|
||||
<ul>
|
||||
<li>End-to-end encryption (TLS 1.3)</li>
|
||||
<li>GDPR-compliant data handling</li>
|
||||
<li>Role-based access control (RBAC)</li>
|
||||
<li>Audit logging of all data access</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- System Architecture -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">🏗️ System Architecture</h2>
|
||||
<div class="diagram-container">
|
||||
<svg viewBox="0 0 1400 700">
|
||||
<defs>
|
||||
<marker id="arrow-arch" markerWidth="10" markerHeight="10" refX="9" refY="3" orient="auto">
|
||||
<polygon points="0 0, 10 3, 0 6" fill="#2d3748"/>
|
||||
</marker>
|
||||
</defs>
|
||||
|
||||
<!-- Layer 1: Data Sources -->
|
||||
<text x="50" y="40" fill="#2d3748" font-size="16" font-weight="bold">Layer 1: Data Sources</text>
|
||||
<rect x="50" y="60" width="180" height="100" rx="10" fill="#4299e1" stroke="#2b6cb0" stroke-width="2"/>
|
||||
<text x="140" y="95" text-anchor="middle" fill="white" font-size="12" font-weight="bold">CRM API</text>
|
||||
<text x="140" y="115" text-anchor="middle" fill="white" font-size="10">Salesforce REST</text>
|
||||
<text x="140" y="130" text-anchor="middle" fill="white" font-size="10">OAuth 2.0</text>
|
||||
|
||||
<rect x="260" y="60" width="180" height="100" rx="10" fill="#4299e1" stroke="#2b6cb0" stroke-width="2"/>
|
||||
<text x="350" y="95" text-anchor="middle" fill="white" font-size="12" font-weight="bold">Orders DB</text>
|
||||
<text x="350" y="115" text-anchor="middle" fill="white" font-size="10">MySQL 8.0</text>
|
||||
<text x="350" y="130" text-anchor="middle" fill="white" font-size="10">Read replica</text>
|
||||
|
||||
<rect x="470" y="60" width="180" height="100" rx="10" fill="#4299e1" stroke="#2b6cb0" stroke-width="2"/>
|
||||
<text x="560" y="95" text-anchor="middle" fill="white" font-size="12" font-weight="bold">CSV Files</text>
|
||||
<text x="560" y="115" text-anchor="middle" fill="white" font-size="10">S3 Bucket</text>
|
||||
<text x="560" y="130" text-anchor="middle" fill="white" font-size="10">Daily exports</text>
|
||||
|
||||
<!-- Layer 2: Processing -->
|
||||
<text x="50" y="220" fill="#2d3748" font-size="16" font-weight="bold">Layer 2: Processing</text>
|
||||
<rect x="50" y="240" width="250" height="100" rx="10" fill="#ed8936" stroke="#c05621" stroke-width="2"/>
|
||||
<text x="175" y="275" text-anchor="middle" fill="white" font-size="12" font-weight="bold">Airflow DAGs</text>
|
||||
<text x="175" y="295" text-anchor="middle" fill="white" font-size="10">Python 3.11</text>
|
||||
<text x="175" y="310" text-anchor="middle" fill="white" font-size="10">Orchestration & scheduling</text>
|
||||
|
||||
<rect x="330" y="240" width="250" height="100" rx="10" fill="#ed8936" stroke="#c05621" stroke-width="2"/>
|
||||
<text x="455" y="275" text-anchor="middle" fill="white" font-size="12" font-weight="bold">Lambda Functions</text>
|
||||
<text x="455" y="295" text-anchor="middle" fill="white" font-size="10">Node.js 20</text>
|
||||
<text x="455" y="310" text-anchor="middle" fill="white" font-size="10">Data transformations</text>
|
||||
|
||||
<!-- Layer 3: External Services -->
|
||||
<text x="50" y="400" fill="#2d3748" font-size="16" font-weight="bold">Layer 3: External Services</text>
|
||||
<rect x="50" y="420" width="220" height="100" rx="10" fill="#9f7aea" stroke="#6b46c1" stroke-width="2"/>
|
||||
<text x="160" y="455" text-anchor="middle" fill="white" font-size="12" font-weight="bold">Geo API</text>
|
||||
<text x="160" y="475" text-anchor="middle" fill="white" font-size="10">Location enrichment</text>
|
||||
<text x="160" y="490" text-anchor="middle" fill="white" font-size="10">MaxMind GeoIP2</text>
|
||||
|
||||
<rect x="300" y="420" width="220" height="100" rx="10" fill="#9f7aea" stroke="#6b46c1" stroke-width="2"/>
|
||||
<text x="410" y="455" text-anchor="middle" fill="white" font-size="12" font-weight="bold">Clearbit</text>
|
||||
<text x="410" y="475" text-anchor="middle" fill="white" font-size="10">Company data</text>
|
||||
<text x="410" y="490" text-anchor="middle" fill="white" font-size="10">Firmographics API</text>
|
||||
|
||||
<!-- Layer 4: Storage -->
|
||||
<text x="50" y="580" fill="#2d3748" font-size="16" font-weight="bold">Layer 4: Output & Storage</text>
|
||||
<rect x="50" y="600" width="250" height="80" rx="10" fill="#48bb78" stroke="#2f855a" stroke-width="2"/>
|
||||
<text x="175" y="635" text-anchor="middle" fill="white" font-size="12" font-weight="bold">BigQuery</text>
|
||||
<text x="175" y="655" text-anchor="middle" fill="white" font-size="10">Data warehouse</text>
|
||||
|
||||
<rect x="330" y="600" width="250" height="80" rx="10" fill="#48bb78" stroke="#2f855a" stroke-width="2"/>
|
||||
<text x="455" y="635" text-anchor="middle" fill="white" font-size="12" font-weight="bold">Redis Cache</text>
|
||||
<text x="455" y="655" text-anchor="middle" fill="white" font-size="10">Query acceleration</text>
|
||||
|
||||
<!-- Arrows -->
|
||||
<path d="M 140 160 L 175 240" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-arch)"/>
|
||||
<path d="M 350 160 L 300 240" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-arch)"/>
|
||||
<path d="M 175 340 L 160 420" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-arch)"/>
|
||||
<path d="M 175 340 L 175 600" stroke="#2d3748" stroke-width="2" marker-end="url(#arrow-arch)"/>
|
||||
|
||||
<!-- Supporting info -->
|
||||
<rect x="750" y="240" width="600" height="260" rx="15" fill="#f7fafc" stroke="#cbd5e0" stroke-width="2"/>
|
||||
<text x="1050" y="275" text-anchor="middle" fill="#2d3748" font-size="14" font-weight="bold">Technology Stack</text>
|
||||
|
||||
<text x="780" y="310" fill="#2d3748" font-size="11" font-weight="bold">Languages & Frameworks:</text>
|
||||
<text x="780" y="330" fill="#4a5568" font-size="10">• Python 3.11 (data processing)</text>
|
||||
<text x="780" y="345" fill="#4a5568" font-size="10">• Node.js 20 (Lambda functions)</text>
|
||||
<text x="780" y="360" fill="#4a5568" font-size="10">• SQL (data queries)</text>
|
||||
|
||||
<text x="780" y="390" fill="#2d3748" font-size="11" font-weight="bold">AWS Services:</text>
|
||||
<text x="780" y="410" fill="#4a5568" font-size="10">• Lambda (serverless compute)</text>
|
||||
<text x="780" y="425" fill="#4a5568" font-size="10">• S3 (object storage)</text>
|
||||
<text x="780" y="440" fill="#4a5568" font-size="10">• CloudWatch (monitoring)</text>
|
||||
<text x="780" y="455" fill="#4a5568" font-size="10">• IAM (access control)</text>
|
||||
|
||||
<text x="780" y="480" fill="#2d3748" font-size="11" font-weight="bold">Dependencies:</text>
|
||||
<text x="780" y="500" fill="#4a5568" font-size="10">• pandas, SQLAlchemy (Python)</text>
|
||||
</svg>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Deployment -->
|
||||
<div class="section">
|
||||
<h2 class="section-title">🚀 Deployment & Usage</h2>
|
||||
<div class="feature-grid">
|
||||
<div class="feature-card">
|
||||
<h3>Deployment Model</h3>
|
||||
<ul>
|
||||
<li>Cloud-hosted (AWS)</li>
|
||||
<li>Serverless architecture</li>
|
||||
<li>Multi-region for redundancy</li>
|
||||
<li>Infrastructure as Code (Terraform)</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Prerequisites</h3>
|
||||
<ul>
|
||||
<li>AWS account with appropriate IAM roles</li>
|
||||
<li>Salesforce API credentials</li>
|
||||
<li>MySQL read replica access</li>
|
||||
<li>BigQuery project setup</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="feature-card">
|
||||
<h3>Typical Workflow</h3>
|
||||
<ul>
|
||||
<li>1. Configure data source connections</li>
|
||||
<li>2. Deploy Airflow DAGs</li>
|
||||
<li>3. Run initial backfill</li>
|
||||
<li>4. Monitor daily incremental runs</li>
|
||||
<li>5. Query unified data in BigQuery</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Legend -->
|
||||
<div class="legend">
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #4299e1;"></div>
|
||||
<span>Data Sources</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #ed8936;"></div>
|
||||
<span>Processing Logic</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #9f7aea;"></div>
|
||||
<span>External Services</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #48bb78;"></div>
|
||||
<span>Output/Storage</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #38b2ac;"></div>
|
||||
<span>Data Quality</span>
|
||||
</div>
|
||||
<div class="legend-item">
|
||||
<div class="legend-box" style="background: #805ad5;"></div>
|
||||
<span>Enrichment</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<footer>
|
||||
<strong>DataFlow ETL Pipeline Architecture v1.0</strong><br>
|
||||
Generated: 2025-11-03 | Customer Data Integration Platform<br>
|
||||
Technologies: Python, Node.js, AWS Lambda, Apache Airflow, BigQuery, MySQL
|
||||
</footer>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user