Initial commit
This commit is contained in:
Zhongwei Li
321
skills/references/java.md
Normal file
321
skills/references/java.md
Normal file
@@ -0,0 +1,321 @@
|
||||
# Java Architecture Patterns
|
||||
|
||||
## Table of Contents
|
||||
- Common Frameworks and Patterns
|
||||
- Project Structure Conventions
|
||||
- Deployment Considerations
|
||||
- Best Practices
|
||||
|
||||
## Common Frameworks and Patterns
|
||||
|
||||
### Spring Boot
|
||||
**Typical Structure:**
|
||||
```
|
||||
project/
|
||||
├── src/main/java/com/company/project/
|
||||
│ ├── Application.java # Main class
|
||||
│ ├── controller/ # REST controllers
|
||||
│ ├── service/ # Business logic
|
||||
│ ├── repository/ # Data access
|
||||
│ ├── model/entity/ # JPA entities
|
||||
│ ├── model/dto/ # DTOs
|
||||
│ ├── config/ # Configuration
|
||||
│ └── exception/ # Exception handling
|
||||
├── src/main/resources/
|
||||
│ ├── application.properties
|
||||
│ └── application-{env}.properties
|
||||
├── src/test/java/
|
||||
└── pom.xml or build.gradle
|
||||
```
|
||||
|
||||
**Key Characteristics:**
|
||||
- Convention over configuration
|
||||
- Auto-configuration
|
||||
- Embedded servers (Tomcat, Jetty)
|
||||
- Spring ecosystem integration
|
||||
- Dependency injection
|
||||
|
||||
### Jakarta EE (formerly Java EE)
|
||||
**Key Technologies:**
|
||||
- JAX-RS for REST APIs
|
||||
- JPA for persistence
|
||||
- CDI for dependency injection
|
||||
- EJB for business logic
|
||||
- Deployed to application servers (WildFly, Payara)
|
||||
|
||||
### Micronaut
|
||||
**Key Characteristics:**
|
||||
- Fast startup, low memory
|
||||
- Compile-time dependency injection
|
||||
- Native image support (GraalVM)
|
||||
- Cloud-native focus
|
||||
|
||||
### Quarkus
|
||||
**Key Characteristics:**
|
||||
- Kubernetes-native
|
||||
- Fast startup, low memory
|
||||
- Developer experience focused
|
||||
- GraalVM native images
|
||||
|
||||
## Project Structure Conventions
|
||||
|
||||
### Spring Boot Microservice
|
||||
```
|
||||
project/
|
||||
├── src/main/java/com/company/service/
|
||||
│ ├── ServiceApplication.java
|
||||
│ ├── api/ # API layer
|
||||
│ │ ├── controller/
|
||||
│ │ └── dto/
|
||||
│ ├── domain/ # Domain layer
|
||||
│ │ ├── model/
|
||||
│ │ └── service/
|
||||
│ ├── infrastructure/ # Infrastructure
|
||||
│ │ ├── repository/
|
||||
│ │ ├── client/
|
||||
│ │ └── config/
|
||||
│ └── common/ # Shared utilities
|
||||
├── src/main/resources/
|
||||
├── src/test/
|
||||
├── pom.xml
|
||||
└── Dockerfile
|
||||
```
|
||||
|
||||
### Maven Multi-Module Project
|
||||
```
|
||||
parent-project/
|
||||
├── pom.xml # Parent POM
|
||||
├── api-module/
|
||||
│ ├── pom.xml
|
||||
│ └── src/
|
||||
├── service-module/
|
||||
│ ├── pom.xml
|
||||
│ └── src/
|
||||
├── common-module/
|
||||
│ ├── pom.xml
|
||||
│ └── src/
|
||||
└── integration-tests/
|
||||
├── pom.xml
|
||||
└── src/
|
||||
```
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
### Build Tools
|
||||
- **Maven**: XML-based, widespread
|
||||
- **Gradle**: Groovy/Kotlin DSL, flexible, faster
|
||||
|
||||
### Application Servers
|
||||
- **Spring Boot**: Embedded Tomcat/Jetty/Undertow
|
||||
- **Traditional**: WildFly, Payara, WebLogic, WebSphere
|
||||
- **Lightweight**: Tomcat, Jetty as standalone
|
||||
|
||||
### Containerization
|
||||
```dockerfile
|
||||
# Multi-stage build
|
||||
FROM maven:3.9-eclipse-temurin-17 AS build
|
||||
WORKDIR /app
|
||||
COPY pom.xml .
|
||||
COPY src ./src
|
||||
RUN mvn clean package -DskipTests
|
||||
|
||||
FROM eclipse-temurin:17-jre-alpine
|
||||
WORKDIR /app
|
||||
COPY --from=build /app/target/*.jar app.jar
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["java", "-jar", "app.jar"]
|
||||
```
|
||||
|
||||
### Common Deployment Platforms
|
||||
- **AWS**: Elastic Beanstalk, ECS, Lambda (with GraalVM)
|
||||
- **Google Cloud**: App Engine, Cloud Run, GKE
|
||||
- **Azure**: App Service, Container Instances, AKS
|
||||
- **Kubernetes**: Primary choice for microservices
|
||||
- **Traditional**: On-premise application servers
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Code Organization
|
||||
|
||||
**Layered Architecture:**
|
||||
- Controller → Service → Repository
|
||||
- DTOs for data transfer
|
||||
- Entities for persistence
|
||||
- Use interfaces for services
|
||||
|
||||
**Package by Feature:**
|
||||
```
|
||||
com.company.project/
|
||||
├── user/
|
||||
│ ├── UserController.java
|
||||
│ ├── UserService.java
|
||||
│ ├── UserRepository.java
|
||||
│ └── User.java
|
||||
├── order/
|
||||
└── product/
|
||||
```
|
||||
|
||||
### Database Integration
|
||||
|
||||
**Spring Data JPA:**
|
||||
```java
|
||||
public interface UserRepository extends JpaRepository<User, Long> {
|
||||
Optional<User> findByEmail(String email);
|
||||
}
|
||||
```
|
||||
|
||||
**Hibernate:** JPA implementation
|
||||
**Flyway/Liquibase:** Database migrations
|
||||
**Connection Pooling:** HikariCP (default in Spring Boot)
|
||||
|
||||
### Configuration
|
||||
|
||||
**Spring Boot application.properties:**
|
||||
```properties
|
||||
spring.datasource.url=${DB_URL}
|
||||
spring.jpa.hibernate.ddl-auto=validate
|
||||
server.port=${PORT:8080}
|
||||
```
|
||||
|
||||
**Profiles:** application-dev.properties, application-prod.properties
|
||||
**Environment variables:** For secrets and env-specific config
|
||||
**Spring Cloud Config:** Centralized configuration
|
||||
|
||||
### Security
|
||||
|
||||
**Spring Security:**
|
||||
- Authentication and authorization
|
||||
- JWT support
|
||||
- OAuth2/OpenID Connect
|
||||
- Method-level security
|
||||
|
||||
**Best Practices:**
|
||||
- Use BCrypt for passwords
|
||||
- Validate input with Bean Validation
|
||||
- Prevent SQL injection (use JPA)
|
||||
- CORS configuration
|
||||
- HTTPS in production
|
||||
|
||||
### Error Handling
|
||||
|
||||
**Spring Boot:**
|
||||
```java
|
||||
@ControllerAdvice
|
||||
public class GlobalExceptionHandler {
|
||||
@ExceptionHandler(ResourceNotFoundException.class)
|
||||
public ResponseEntity<ErrorResponse> handleNotFound(
|
||||
ResourceNotFoundException ex) {
|
||||
return ResponseEntity.status(404)
|
||||
.body(new ErrorResponse(ex.getMessage()));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
**Frameworks:**
|
||||
- **JUnit 5**: Standard testing framework
|
||||
- **Mockito**: Mocking framework
|
||||
- **Spring Boot Test**: Integration testing
|
||||
- **RestAssured**: API testing
|
||||
- **TestContainers**: Database testing with containers
|
||||
|
||||
**Patterns:**
|
||||
- Unit tests with Mockito
|
||||
- Integration tests with @SpringBootTest
|
||||
- Slice tests (@WebMvcTest, @DataJpaTest)
|
||||
- Test coverage with JaCoCo
|
||||
|
||||
### Logging
|
||||
|
||||
**SLF4J + Logback (Spring Boot default):**
|
||||
```java
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
private static final Logger log =
|
||||
LoggerFactory.getLogger(MyClass.class);
|
||||
|
||||
log.info("Processing request for user {}", userId);
|
||||
```
|
||||
|
||||
**Structured Logging:** Logstash encoder for JSON logs
|
||||
|
||||
### API Documentation
|
||||
|
||||
**OpenAPI/Swagger:**
|
||||
- springdoc-openapi for automatic generation
|
||||
- Annotations for customization
|
||||
- Swagger UI for interactive docs
|
||||
|
||||
### Dependency Injection
|
||||
|
||||
**Spring:**
|
||||
- Constructor injection (preferred)
|
||||
- @Autowired annotation
|
||||
- @Component, @Service, @Repository stereotypes
|
||||
- Java Config with @Configuration and @Bean
|
||||
|
||||
### Common Libraries
|
||||
|
||||
- **Lombok**: Reduce boilerplate
|
||||
- **MapStruct**: Object mapping
|
||||
- **Apache Commons**: Utility libraries
|
||||
- **Guava**: Google's core libraries
|
||||
- **Jackson**: JSON processing
|
||||
|
||||
## Technology-Specific Patterns
|
||||
|
||||
### Microservices
|
||||
|
||||
**Spring Cloud:**
|
||||
- Service Discovery: Eureka
|
||||
- API Gateway: Spring Cloud Gateway
|
||||
- Config: Spring Cloud Config
|
||||
- Circuit Breaker: Resilience4j
|
||||
- Distributed Tracing: Micrometer + Zipkin
|
||||
|
||||
### Messaging
|
||||
|
||||
- **Spring AMQP**: RabbitMQ integration
|
||||
- **Spring Kafka**: Kafka integration
|
||||
- **Spring JMS**: JMS support
|
||||
- **AWS SQS**: Cloud messaging
|
||||
|
||||
### Caching
|
||||
|
||||
- **Spring Cache**: Abstraction layer
|
||||
- **Redis**: Distributed cache
|
||||
- **Caffeine**: In-memory cache
|
||||
- **Hazelcast**: Distributed cache
|
||||
|
||||
### Reactive Programming
|
||||
|
||||
**Spring WebFlux:**
|
||||
- Non-blocking, event-driven
|
||||
- Project Reactor
|
||||
- Better for high concurrency
|
||||
- MongoDB/Cassandra reactive support
|
||||
|
||||
### GraphQL
|
||||
|
||||
- **Spring for GraphQL**: Official Spring support
|
||||
- Schema-first approach
|
||||
- DataFetchers for resolvers
|
||||
|
||||
### Monitoring
|
||||
|
||||
- **Spring Boot Actuator**: Health checks, metrics
|
||||
- **Micrometer**: Metrics facade
|
||||
- **Prometheus**: Metrics collection
|
||||
- **Grafana**: Visualization
|
||||
- **Zipkin/Jaeger**: Distributed tracing
|
||||
- **ELK Stack**: Logging
|
||||
|
||||
### Performance
|
||||
|
||||
- **Connection pooling**: HikariCP
|
||||
- **Query optimization**: Proper JPA fetch strategies
|
||||
- **Caching**: Redis, Caffeine
|
||||
- **Async processing**: @Async, CompletableFuture
|
||||
- **JVM tuning**: Heap size, GC configuration
|
||||
431
skills/references/mermaid-diagrams.md
Normal file
431
skills/references/mermaid-diagrams.md
Normal file
@@ -0,0 +1,431 @@
|
||||
# Generate 5 Essential Architecture Diagrams in Mermaid.js
|
||||
|
||||
## Including C4 Level 1, 2, 3, Data Flow, and C4 Deployment
|
||||
|
||||
---
|
||||
|
||||
Goal: generate **5 essential architecture diagrams** in Mermaid.js format based on the provided system information.
|
||||
|
||||
---
|
||||
|
||||
## Instructions
|
||||
|
||||
Generate exactly **5 diagrams** in the following order. Each diagram should be complete, syntactically valid Mermaid code ready to render.
|
||||
|
||||
### Diagram 1: System Context Diagram (C4 Level 1)
|
||||
|
||||
**Purpose**: Show the big picture - your system in context with external users, systems, and services it connects to.
|
||||
|
||||
**Output Format**: Use C4Context Mermaid syntax
|
||||
|
||||
**Requirements**:
|
||||
|
||||
- Show the primary software system as the central element
|
||||
- Include all external users/actors who interact with the system
|
||||
- Include all external systems/services the system depends on
|
||||
- Label all relationships with interaction descriptions
|
||||
- Use clear, descriptive titles and descriptions
|
||||
|
||||
**Mermaid Syntax Template**:
|
||||
|
||||
```
|
||||
C4Context
|
||||
title System Context - [System Name]
|
||||
Person(customer, "Customer", "End user of the system")
|
||||
System(system, "[System Name]", "Core business functionality")
|
||||
System_Ext(external, "External System", "Third-party integration")
|
||||
Rel(customer, system, "uses")
|
||||
Rel(system, external, "calls API")
|
||||
```
|
||||
|
||||
**Key Elements to Include**:
|
||||
|
||||
- At least 1-2 external users
|
||||
- The primary system (your product)
|
||||
- 2-3 external systems/services it depends on
|
||||
|
||||
---
|
||||
|
||||
### Diagram 2: Container Diagram (C4 Level 2)
|
||||
|
||||
**Purpose**: Zoom into the primary system and show its main building blocks (containers) like web apps, APIs, databases, message queues, etc.
|
||||
|
||||
**Output Format**: Use C4Container Mermaid syntax
|
||||
|
||||
**Requirements**:
|
||||
|
||||
- Break down the system into logical containers (applications, services, databases)
|
||||
- Show technology choices for each container (e.g., "Node.js API", "PostgreSQL")
|
||||
- Show interactions between containers with labels
|
||||
- Keep external systems visible but less detailed
|
||||
- Include brief descriptions of what each container does
|
||||
|
||||
**Mermaid Syntax Template**:
|
||||
|
||||
```
|
||||
C4Container
|
||||
title Container Diagram - [System Name]
|
||||
Person(user, "User", "System end user")
|
||||
|
||||
System_Boundary(system, "[System Name]") {
|
||||
Container(web, "Web Application", "React", "Serves the UI")
|
||||
Container(api, "API Server", "Node.js", "Provides API services")
|
||||
Container(db, "Database", "PostgreSQL", "Stores application data")
|
||||
Container(cache, "Cache", "Redis", "Session and data caching")
|
||||
}
|
||||
|
||||
System_Ext(external, "External Service", "Third-party API")
|
||||
|
||||
Rel(user, web, "Uses [HTTPS]")
|
||||
Rel(web, api, "Calls [REST API]")
|
||||
Rel(api, db, "Reads/Writes [SQL]")
|
||||
Rel(api, cache, "Reads/Writes")
|
||||
Rel(api, external, "Calls [REST API]")
|
||||
```
|
||||
|
||||
**Key Elements to Include**:
|
||||
|
||||
- 3-5 main containers (apps, APIs, databases, services)
|
||||
- Technology stack for each container
|
||||
- Container-to-container relationships
|
||||
|
||||
---
|
||||
|
||||
### Diagram 3: Component Diagram (C4 Level 3)
|
||||
|
||||
**Purpose**: Zoom into a specific container and show its internal components/structure (controllers, services, repositories, etc.).
|
||||
|
||||
**Output Format**: Use C4Component Mermaid syntax
|
||||
|
||||
**Requirements**:
|
||||
|
||||
- Focus on ONE important container (usually the API or main service)
|
||||
- Show internal components/classes/modules within that container
|
||||
- Show technology/framework for each component
|
||||
- Show dependencies between components
|
||||
- Include clear descriptions of each component's responsibility
|
||||
- Keep it focused - don't show every single component, show the main 5-8
|
||||
|
||||
**Mermaid Syntax Template**:
|
||||
|
||||
```
|
||||
C4Component
|
||||
title Component Diagram - [Container Name]
|
||||
|
||||
Container_Boundary(container, "[Container Name]") {
|
||||
Component(ctrl, "API Controller", "Express Router", "Handles HTTP requests")
|
||||
Component(service, "Business Service", "Service Class", "Contains business logic")
|
||||
Component(repo, "Data Repository", "Data Access Layer", "Database queries")
|
||||
Component(auth, "Auth Component", "JWT/OAuth", "Handles authentication")
|
||||
Component(email, "Email Component", "Email Service", "Sends emails")
|
||||
}
|
||||
|
||||
ContainerDb(db, "Database", "PostgreSQL", "Persistence")
|
||||
System_Ext(email_service, "Email Service", "SendGrid/SMTP")
|
||||
|
||||
Rel(ctrl, service, "Uses")
|
||||
Rel(service, repo, "Uses")
|
||||
Rel(service, auth, "Uses")
|
||||
Rel(service, email, "Uses")
|
||||
Rel(repo, db, "Reads/Writes")
|
||||
Rel(email, email_service, "Sends via")
|
||||
```
|
||||
|
||||
**Key Elements to Include**:
|
||||
|
||||
- 5-8 main internal components
|
||||
- Component responsibilities clearly described
|
||||
- Dependencies between components
|
||||
- External dependencies (databases, services)
|
||||
- Focus on one container (usually API/backend service)
|
||||
|
||||
---
|
||||
|
||||
### Diagram 4: Data Flow Diagram (DFD)
|
||||
|
||||
**Purpose**: Show how data moves through the system - from sources, through processes and transformations, to storage and destinations.
|
||||
|
||||
**Output Format**: Use flowchart syntax with data process symbols
|
||||
|
||||
**Requirements**:
|
||||
|
||||
- Identify all data sources (users, external systems)
|
||||
- Show major data processing/transformation steps
|
||||
- Show all data storage locations (databases, caches, files)
|
||||
- Show data destinations (output systems, reports, notifications)
|
||||
- Label flows with data type descriptions
|
||||
- Use consistent formatting for similar element types
|
||||
|
||||
**Mermaid Syntax Template**:
|
||||
|
||||
```
|
||||
flowchart LR
|
||||
subgraph sources["📥 Data Sources"]
|
||||
user["👤 User Input"]
|
||||
external["🔗 External APIs"]
|
||||
end
|
||||
|
||||
subgraph processes["⚙️ Data Processing"]
|
||||
validation["Validate Data"]
|
||||
transform["Transform Data"]
|
||||
aggregate["Aggregate Data"]
|
||||
end
|
||||
|
||||
subgraph storage["💾 Data Storage"]
|
||||
db["Primary DB<br/>(PostgreSQL)"]
|
||||
cache["Cache<br/>(Redis)"]
|
||||
archive["Archive<br/>(S3)"]
|
||||
end
|
||||
|
||||
subgraph outputs["📤 Data Outputs"]
|
||||
api_out["API Response"]
|
||||
report["Reports"]
|
||||
notification["Notifications"]
|
||||
end
|
||||
|
||||
user -->|"raw input"| validation
|
||||
external -->|"API data"| validation
|
||||
validation -->|"valid data"| transform
|
||||
transform -->|"structured data"| aggregate
|
||||
aggregate -->|"final data"| db
|
||||
aggregate -->|"hot data"| cache
|
||||
db -->|"queries"| api_out
|
||||
cache -->|"session data"| api_out
|
||||
db -->|"historical"| report
|
||||
aggregate -->|"events"| notification
|
||||
db -->|"old data"| archive
|
||||
```
|
||||
|
||||
**Key Elements to Include**:
|
||||
|
||||
- Data sources (at least 2)
|
||||
- Data transformation steps (at least 3)
|
||||
- Data storage locations
|
||||
- Data destinations
|
||||
- Flow labels showing data types
|
||||
|
||||
---
|
||||
|
||||
### Diagram 5: Deployment Diagram (C4Deployment)
|
||||
|
||||
**Purpose**: Show how software components are deployed across physical/cloud infrastructure, environments, and how they're replicated/distributed.
|
||||
|
||||
**Output Format**: Use C4Deployment Mermaid syntax with Deployment_Node and Node elements
|
||||
|
||||
**Requirements**:
|
||||
|
||||
- Show deployment environments (production, staging)
|
||||
- Show cloud providers or infrastructure boundaries
|
||||
- Show servers/nodes/instances where containers run
|
||||
- Show load balancers, databases, and other infrastructure
|
||||
- Show internal connections between nodes
|
||||
- Indicate scaling and high availability configurations
|
||||
- Use proper C4Deployment syntax with Deployment_Node, Node, and Rel
|
||||
|
||||
**Mermaid Syntax Template**:
|
||||
|
||||
```mermaid
|
||||
C4Deployment
|
||||
title <Diagram Title>
|
||||
Deployment_Node(<id>, "<Name>", "<Technology/Environment>") {
|
||||
Container(<id>, "<Name>", "<Technology>", "<Description>")
|
||||
}
|
||||
|
||||
Deployment_Node(<id>, "<Name>", "<Technology/Environment>") {
|
||||
Deployment_Node(<id>, "<Name>", "<Technology>") {
|
||||
Container(<id>, "<Name>", "<Technology>", "<Description>")
|
||||
}
|
||||
}
|
||||
|
||||
Deployment_Node(<id>, "<Name>", "<Technology/Environment>") {
|
||||
Deployment_Node(<id>, "<Name>", "<Technology>") {
|
||||
ContainerDb(<id>, "<Name>", "<Technology>", "<Description>")
|
||||
}
|
||||
}
|
||||
|
||||
Rel(<source_id>, <target_id>, "<Description>", "<Protocol>")
|
||||
Rel_U(<source_id>, <target_id>, "<Description>")
|
||||
Rel_R(<source_id>, <target_id>, "<Description>")
|
||||
|
||||
UpdateRelStyle(<source_id>, <target_id>, $offsetX="<value>", $offsetY="<value>")
|
||||
|
||||
```
|
||||
|
||||
**Notes:**
|
||||
- Use Deployment_Node for devices, servers, or environments.
|
||||
- Use Container for applications or services.
|
||||
- Use ContainerDb for databases.
|
||||
- Use Rel, Rel_U, or Rel_R for relationships.
|
||||
- Use UpdateRelStyle to adjust label or arrow positions.
|
||||
|
||||
## Example: C4Deployment for Internet Banking System
|
||||
|
||||
```mermaid
|
||||
C4Deployment
|
||||
title Deployment Diagram for Internet Banking System - Live
|
||||
|
||||
Deployment_Node(mob, "Customer's mobile device", "Apple IOS or Android"){
|
||||
Container(mobile, "Mobile App", "Xamarin", "Provides a limited subset of the Internet Banking functionality to customers via their mobile device.")
|
||||
}
|
||||
|
||||
Deployment_Node(comp, "Customer's computer", "Microsoft Windows or Apple macOS"){
|
||||
Deployment_Node(browser, "Web Browser", "Google Chrome, Mozilla Firefox,<br/> Apple Safari or Microsoft Edge"){
|
||||
Container(spa, "Single Page Application", "JavaScript and Angular", "Provides all of the Internet Banking functionality to customers via their web browser.")
|
||||
}
|
||||
}
|
||||
|
||||
Deployment_Node(plc, "Big Bank plc", "Big Bank plc data center"){
|
||||
Deployment_Node(dn, "bigbank-api*** x8", "Ubuntu 16.04 LTS"){
|
||||
Deployment_Node(apache, "Apache Tomcat", "Apache Tomcat 8.x"){
|
||||
Container(api, "API Application", "Java and Spring MVC", "Provides Internet Banking functionality via a JSON/HTTPS API.")
|
||||
}
|
||||
}
|
||||
Deployment_Node(bb2, "bigbank-web*** x4", "Ubuntu 16.04 LTS"){
|
||||
Deployment_Node(apache2, "Apache Tomcat", "Apache Tomcat 8.x"){
|
||||
Container(web, "Web Application", "Java and Spring MVC", "Delivers the static content and the Internet Banking single page application.")
|
||||
}
|
||||
}
|
||||
Deployment_Node(bigbankdb01, "bigbank-db01", "Ubuntu 16.04 LTS"){
|
||||
Deployment_Node(oracle, "Oracle - Primary", "Oracle 12c"){
|
||||
ContainerDb(db, "Database", "Relational Database Schema", "Stores user registration information, hashed authentication credentials, access logs, etc.")
|
||||
}
|
||||
}
|
||||
Deployment_Node(bigbankdb02, "bigbank-db02", "Ubuntu 16.04 LTS") {
|
||||
Deployment_Node(oracle2, "Oracle - Secondary", "Oracle 12c") {
|
||||
ContainerDb(db2, "Database", "Relational Database Schema", "Stores user registration information, hashed authentication credentials, access logs, etc.")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Rel(mobile, api, "Makes API calls to", "json/HTTPS")
|
||||
Rel(spa, api, "Makes API calls to", "json/HTTPS")
|
||||
Rel_U(web, spa, "Delivers to the customer's web browser")
|
||||
Rel(api, db, "Reads from and writes to", "JDBC")
|
||||
Rel(api, db2, "Reads from and writes to", "JDBC")
|
||||
Rel_R(db, db2, "Replicates data to")
|
||||
|
||||
UpdateRelStyle(spa, api, $offsetY="-40")
|
||||
UpdateRelStyle(web, spa, $offsetY="-40")
|
||||
UpdateRelStyle(api, db, $offsetY="-20", $offsetX="5")
|
||||
UpdateRelStyle(api, db2, $offsetX="-40", $offsetY="-20")
|
||||
UpdateRelStyle(db, db2, $offsetY="-10")
|
||||
|
||||
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Key Elements to Include**:
|
||||
|
||||
- Cloud provider/infrastructure boundary
|
||||
- Production environment with load balancer
|
||||
- Multiple instances of compute nodes for high availability
|
||||
- Database with read replicas
|
||||
- Caching layer
|
||||
- External service connections
|
||||
- Message queues for async processing
|
||||
|
||||
---
|
||||
|
||||
## Input Information Required
|
||||
|
||||
Before generating the diagrams, gather or confirm the following information about the system:
|
||||
|
||||
### Project Overview
|
||||
|
||||
- System name and purpose
|
||||
- Primary users/actors
|
||||
- Main external systems it integrates with
|
||||
|
||||
### Technical Components (Containers)
|
||||
|
||||
- List all applications/services in the system
|
||||
- Technology stack for each container (language, framework)
|
||||
- Database systems and their purposes
|
||||
- Message queues, caches, or other data stores
|
||||
- Third-party services (payment, auth, notifications, etc.)
|
||||
|
||||
### Internal Components (for Level 3)
|
||||
|
||||
- Main controller/handler types
|
||||
- Key service/business logic components
|
||||
- Data access/repository layers
|
||||
- Utility/helper components
|
||||
- Authentication/authorization components
|
||||
|
||||
### Data Processing
|
||||
|
||||
- Main data types flowing through the system
|
||||
- How data enters the system (APIs, uploads, imports)
|
||||
- Major processing/transformation steps
|
||||
- How data is stored and how long
|
||||
- How data exits the system (exports, reports, APIs)
|
||||
|
||||
### Infrastructure & Deployment
|
||||
|
||||
- Cloud provider (AWS, GCP, Azure, on-premises)
|
||||
- Number of instances/replicas for high availability
|
||||
- Load balancing strategy
|
||||
- Database replication/backup strategy
|
||||
- CDN or content delivery strategy
|
||||
- Message queue setup
|
||||
|
||||
---
|
||||
|
||||
## Generation Steps
|
||||
|
||||
1. **Understand the context**: Read and understand the system description
|
||||
2. **Generate Diagram 1**: System Context (C4 Level 1) - showing external relationships
|
||||
3. **Generate Diagram 2**: Container (C4 Level 2) - showing internal components and technologies
|
||||
4. **Generate Diagram 3**: Component (C4 Level 3) - showing one container's internal structure
|
||||
5. **Generate Diagram 4**: Data Flow - showing how data moves through the system
|
||||
6. **Generate Diagram 5**: Deployment (C4Deployment) - showing infrastructure and deployment topology
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
For each diagram, output:
|
||||
|
||||
```markdown
|
||||
### Diagram [Number]: [Diagram Title]
|
||||
|
||||
**Description**: [One sentence about what this diagram shows]
|
||||
|
||||
**Focus**: [What this diagram emphasizes]
|
||||
|
||||
\`\`\`mermaid
|
||||
[Complete Mermaid diagram code]
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quality Checklist
|
||||
|
||||
For each diagram, verify:
|
||||
|
||||
- ✅ Syntax is valid Mermaid.js code
|
||||
- ✅ Diagram clearly shows the intended architectural layer
|
||||
- ✅ All relationships/connections are labeled
|
||||
- ✅ Component descriptions are concise and clear
|
||||
- ✅ Technology choices are explicitly shown (where applicable)
|
||||
- ✅ No syntax errors or unclosed elements
|
||||
- ✅ Diagram is readable and not overcrowded
|
||||
- ✅ Follows C4 model conventions for that level
|
||||
- ✅ For C4Deployment: Uses proper Deployment_Node, Node, and Rel syntax
|
||||
|
||||
---
|
||||
|
||||
## Why All 5 Diagrams?
|
||||
|
||||
| Diagram | Audience | Detail Level | Purpose |
|
||||
| ---------------------- | -------------------------------------- | ---------------- | ------------------------------------------ |
|
||||
| **Level 1: Context** | Everyone | Ultra-high level | System boundary and external relationships |
|
||||
| **Level 2: Container** | Tech leads, architects, devs | High level | Main technical components |
|
||||
| **Level 3: Component** | Senior devs, architects | Mid-level | How one container is built internally |
|
||||
| **Data Flow** | Data engineers, backend devs, security | Process level | Data movement and transformations |
|
||||
| **Deployment** | DevOps, SRE, ops team | Infrastructure | How it runs in production |
|
||||
|
||||
---
|
||||
|
||||
355
skills/references/microservices.md
Normal file
355
skills/references/microservices.md
Normal file
@@ -0,0 +1,355 @@
|
||||
# Microservices Architecture Patterns
|
||||
|
||||
## Table of Contents
|
||||
- Core Principles
|
||||
- Service Design Patterns
|
||||
- Communication Patterns
|
||||
- Data Management
|
||||
- Deployment Patterns
|
||||
- Observability
|
||||
|
||||
## Core Principles
|
||||
|
||||
### Single Responsibility
|
||||
Each service owns one business capability and does it well.
|
||||
|
||||
### Decentralized Data
|
||||
Each service has its own database (database-per-service pattern).
|
||||
|
||||
### Independence
|
||||
Services can be developed, deployed, and scaled independently.
|
||||
|
||||
### Resilience
|
||||
Services must handle failures gracefully (circuit breakers, retries).
|
||||
|
||||
## Service Design Patterns
|
||||
|
||||
### Service Boundaries
|
||||
Define services around business capabilities, not technical layers:
|
||||
- ✅ User Service, Order Service, Payment Service
|
||||
- ❌ Database Service, Email Service (too technical)
|
||||
|
||||
### Service Size
|
||||
Guideline: A service should be maintainable by a small team (2-pizza team).
|
||||
|
||||
### API Design
|
||||
- Use RESTful APIs for synchronous communication
|
||||
- Use gRPC for high-performance internal communication
|
||||
- Use message queues for asynchronous communication
|
||||
- Version APIs (v1, v2) to support backward compatibility
|
||||
|
||||
## Communication Patterns
|
||||
|
||||
### Synchronous (Request-Response)
|
||||
|
||||
**REST APIs:**
|
||||
```
|
||||
Service A --HTTP--> Service B
|
||||
<-JSON---
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Need immediate response
|
||||
- Simple request-response flow
|
||||
- External-facing APIs
|
||||
|
||||
**gRPC:**
|
||||
```
|
||||
Service A --Protocol Buffer--> Service B
|
||||
<--Binary Response---
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Internal service communication
|
||||
- Need high performance
|
||||
- Strong typing required
|
||||
|
||||
### Asynchronous (Event-Driven)
|
||||
|
||||
**Message Queue:**
|
||||
```
|
||||
Service A --publish--> Queue --consume--> Service B
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Don't need immediate response
|
||||
- Decouple services
|
||||
- Handle bursts of traffic
|
||||
- Background processing
|
||||
|
||||
**Event Bus/Broker:**
|
||||
```
|
||||
Service A --emit event--> Event Bus --subscribe--> Service B, C, D
|
||||
```
|
||||
|
||||
**When to use:**
|
||||
- Multiple services need same event
|
||||
- Event sourcing patterns
|
||||
- Complex event processing
|
||||
|
||||
**Popular Tools:**
|
||||
- Kafka: High-throughput, event streaming
|
||||
- RabbitMQ: Traditional message broker
|
||||
- AWS SQS/SNS: Cloud-native messaging
|
||||
- NATS: Lightweight messaging
|
||||
|
||||
### API Gateway Pattern
|
||||
|
||||
Centralized entry point for all client requests.
|
||||
|
||||
**Responsibilities:**
|
||||
- Request routing
|
||||
- Authentication/Authorization
|
||||
- Rate limiting
|
||||
- Request/Response transformation
|
||||
- Caching
|
||||
- Load balancing
|
||||
|
||||
**Tools:**
|
||||
- Kong
|
||||
- AWS API Gateway
|
||||
- Nginx
|
||||
- Spring Cloud Gateway
|
||||
- Traefik
|
||||
|
||||
### Service Mesh Pattern
|
||||
|
||||
Infrastructure layer for service-to-service communication.
|
||||
|
||||
**Features:**
|
||||
- Traffic management
|
||||
- Security (mTLS)
|
||||
- Observability
|
||||
- Circuit breaking
|
||||
- Retry logic
|
||||
|
||||
**Tools:**
|
||||
- Istio
|
||||
- Linkerd
|
||||
- Consul Connect
|
||||
|
||||
## Data Management
|
||||
|
||||
### Database Per Service
|
||||
|
||||
Each service owns its database. No shared databases.
|
||||
|
||||
**Benefits:**
|
||||
- Service independence
|
||||
- Technology flexibility
|
||||
- Scalability
|
||||
|
||||
**Challenges:**
|
||||
- Data consistency
|
||||
- Joins across services
|
||||
- Transactions
|
||||
|
||||
### Saga Pattern
|
||||
|
||||
Manage transactions across multiple services.
|
||||
|
||||
**Choreography:**
|
||||
```
|
||||
Order Service --creates order-->
|
||||
|--event--> Payment Service --processes payment-->
|
||||
|--event--> Inventory Service --reserves items-->
|
||||
|--event--> Shipping Service --ships order-->
|
||||
```
|
||||
|
||||
Each service listens to events and publishes new events.
|
||||
|
||||
**Orchestration:**
|
||||
```
|
||||
Order Orchestrator
|
||||
|--call--> Payment Service
|
||||
|--call--> Inventory Service
|
||||
|--call--> Shipping Service
|
||||
```
|
||||
|
||||
Central orchestrator coordinates the saga.
|
||||
|
||||
### CQRS (Command Query Responsibility Segregation)
|
||||
|
||||
Separate read and write models.
|
||||
|
||||
**Pattern:**
|
||||
- Write: Command Model (normalized, transactional)
|
||||
- Read: Query Model (denormalized, optimized for reads)
|
||||
- Sync via events or batch processes
|
||||
|
||||
**When to use:**
|
||||
- Complex domains
|
||||
- Different read/write patterns
|
||||
- Need to scale reads separately
|
||||
|
||||
### Event Sourcing
|
||||
|
||||
Store state changes as events rather than current state.
|
||||
|
||||
**Benefits:**
|
||||
- Complete audit log
|
||||
- Temporal queries
|
||||
- Event replay for debugging
|
||||
- Derive new models from events
|
||||
|
||||
**Challenges:**
|
||||
- Complexity
|
||||
- Event versioning
|
||||
- Storage requirements
|
||||
|
||||
## Deployment Patterns
|
||||
|
||||
### Containerization
|
||||
|
||||
**Docker:**
|
||||
```
|
||||
Each service → Docker image → Container
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Consistency across environments
|
||||
- Isolation
|
||||
- Resource efficiency
|
||||
|
||||
### Container Orchestration
|
||||
|
||||
**Kubernetes:**
|
||||
```
|
||||
Cluster
|
||||
├── Namespace: production
|
||||
│ ├── Deployment: user-service
|
||||
│ ├── Deployment: order-service
|
||||
│ └── Deployment: payment-service
|
||||
└── Namespace: staging
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- Automated deployment
|
||||
- Scaling
|
||||
- Self-healing
|
||||
- Service discovery
|
||||
- Load balancing
|
||||
- Rolling updates
|
||||
|
||||
### Service Discovery
|
||||
|
||||
**Pattern:**
|
||||
Services register themselves and discover other services dynamically.
|
||||
|
||||
**Client-side:**
|
||||
- Service queries registry
|
||||
- Service makes direct call
|
||||
- Tools: Eureka, Consul
|
||||
|
||||
**Server-side:**
|
||||
- Load balancer queries registry
|
||||
- Routes request to service
|
||||
- Tools: Kubernetes, AWS ELB
|
||||
|
||||
### Configuration Management
|
||||
|
||||
**External Configuration:**
|
||||
- Spring Cloud Config
|
||||
- Consul
|
||||
- etcd
|
||||
- Kubernetes ConfigMaps/Secrets
|
||||
|
||||
**Pattern:**
|
||||
- Store config separately from code
|
||||
- Environment-specific configs
|
||||
- Runtime configuration changes
|
||||
|
||||
## Observability
|
||||
|
||||
### Distributed Tracing
|
||||
|
||||
Track requests across multiple services.
|
||||
|
||||
**Tools:**
|
||||
- Jaeger
|
||||
- Zipkin
|
||||
- AWS X-Ray
|
||||
- Datadog APM
|
||||
|
||||
**Pattern:**
|
||||
```
|
||||
Request ID: abc123
|
||||
User Service (10ms) --> Order Service (50ms) --> Payment Service (100ms)
|
||||
Total: 160ms
|
||||
```
|
||||
|
||||
### Centralized Logging
|
||||
|
||||
Aggregate logs from all services.
|
||||
|
||||
**Tools:**
|
||||
- ELK Stack (Elasticsearch, Logstash, Kibana)
|
||||
- Splunk
|
||||
- CloudWatch Logs
|
||||
- Datadog
|
||||
|
||||
**Pattern:**
|
||||
- Structured logging (JSON)
|
||||
- Correlation IDs
|
||||
- Log levels
|
||||
- Searchable aggregation
|
||||
|
||||
### Metrics and Monitoring
|
||||
|
||||
**Metrics to track:**
|
||||
- Request rate
|
||||
- Error rate
|
||||
- Response time (latency)
|
||||
- Resource usage (CPU, memory)
|
||||
|
||||
**Tools:**
|
||||
- Prometheus + Grafana
|
||||
- CloudWatch
|
||||
- Datadog
|
||||
- New Relic
|
||||
|
||||
### Health Checks
|
||||
|
||||
**Types:**
|
||||
- Liveness: Is service alive?
|
||||
- Readiness: Can service handle requests?
|
||||
- Startup: Has service started?
|
||||
|
||||
**Implementation:**
|
||||
```
|
||||
/health/live → 200 OK
|
||||
/health/ready → 200 OK (or 503 if not ready)
|
||||
```
|
||||
|
||||
## Common Challenges
|
||||
|
||||
### Network Latency
|
||||
**Solution:** Async communication, caching, service mesh
|
||||
|
||||
### Data Consistency
|
||||
**Solution:** Eventual consistency, saga pattern, CQRS
|
||||
|
||||
### Testing
|
||||
**Solution:** Contract testing, integration tests, chaos engineering
|
||||
|
||||
### Debugging
|
||||
**Solution:** Distributed tracing, centralized logging, correlation IDs
|
||||
|
||||
### Security
|
||||
**Solution:** API gateway, service mesh mTLS, OAuth2/JWT
|
||||
|
||||
### Service Proliferation
|
||||
**Solution:** Clear service boundaries, API standards, governance
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start with a Monolith:** Don't start with microservices
|
||||
2. **Define Clear Boundaries:** Use Domain-Driven Design
|
||||
3. **Automate Everything:** CI/CD, testing, deployment
|
||||
4. **Monitor from Day One:** Logging, metrics, tracing
|
||||
5. **Design for Failure:** Circuit breakers, retries, timeouts
|
||||
6. **Version APIs:** Support backward compatibility
|
||||
7. **Document APIs:** OpenAPI/Swagger
|
||||
8. **Use Async for Non-Critical Paths:** Message queues
|
||||
9. **Implement Health Checks:** For orchestration
|
||||
10. **Security at Every Layer:** Gateway, service mesh, code
|
||||
244
skills/references/nodejs.md
Normal file
244
skills/references/nodejs.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# Node.js Architecture Patterns
|
||||
|
||||
## Table of Contents
|
||||
- Common Frameworks and Patterns
|
||||
- Project Structure Conventions
|
||||
- Deployment Considerations
|
||||
- Best Practices
|
||||
|
||||
## Common Frameworks and Patterns
|
||||
|
||||
### Express.js
|
||||
**Typical Structure:**
|
||||
```
|
||||
src/
|
||||
├── routes/ # Route definitions
|
||||
├── controllers/ # Request handlers
|
||||
├── services/ # Business logic
|
||||
├── models/ # Data models
|
||||
├── middleware/ # Custom middleware
|
||||
└── config/ # Configuration
|
||||
```
|
||||
|
||||
**Key Characteristics:**
|
||||
- Minimalist, unopinionated framework
|
||||
- Middleware-based architecture
|
||||
- Common with REST APIs
|
||||
- Often paired with ORM (Sequelize, Prisma, TypeORM)
|
||||
|
||||
### NestJS
|
||||
**Typical Structure:**
|
||||
```
|
||||
src/
|
||||
├── modules/ # Feature modules
|
||||
│ └── user/
|
||||
│ ├── user.controller.ts
|
||||
│ ├── user.service.ts
|
||||
│ ├── user.module.ts
|
||||
│ └── dto/
|
||||
├── common/ # Shared utilities
|
||||
└── config/ # Configuration
|
||||
```
|
||||
|
||||
**Key Characteristics:**
|
||||
- Opinionated, TypeScript-first
|
||||
- Dependency injection
|
||||
- Modular architecture
|
||||
- Built-in support for microservices, GraphQL, WebSockets
|
||||
|
||||
### Fastify
|
||||
**Key Characteristics:**
|
||||
- Performance-focused
|
||||
- Schema-based validation
|
||||
- Plugin architecture
|
||||
- Similar patterns to Express but faster
|
||||
|
||||
### Koa
|
||||
**Key Characteristics:**
|
||||
- Lightweight, modern middleware
|
||||
- Async/await native
|
||||
- No bundled middleware
|
||||
- More control than Express
|
||||
|
||||
## Project Structure Conventions
|
||||
|
||||
### REST API (Express/Koa)
|
||||
```
|
||||
project/
|
||||
├── src/
|
||||
│ ├── api/
|
||||
│ │ ├── routes/
|
||||
│ │ └── controllers/
|
||||
│ ├── services/
|
||||
│ ├── models/
|
||||
│ ├── middleware/
|
||||
│ ├── utils/
|
||||
│ └── config/
|
||||
├── tests/
|
||||
├── .env
|
||||
└── package.json
|
||||
```
|
||||
|
||||
### Microservices (NestJS)
|
||||
```
|
||||
project/
|
||||
├── apps/ # Multiple services
|
||||
│ ├── api-gateway/
|
||||
│ ├── user-service/
|
||||
│ └── order-service/
|
||||
├── libs/ # Shared libraries
|
||||
│ ├── common/
|
||||
│ └── database/
|
||||
└── package.json
|
||||
```
|
||||
|
||||
### Serverless (Lambda)
|
||||
```
|
||||
project/
|
||||
├── functions/
|
||||
│ ├── handler1/
|
||||
│ │ └── index.js
|
||||
│ ├── handler2/
|
||||
│ │ └── index.js
|
||||
│ └── shared/
|
||||
├── layers/ # Lambda layers
|
||||
└── serverless.yml
|
||||
```
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
### Containerization
|
||||
**Dockerfile patterns:**
|
||||
- Multi-stage builds for smaller images
|
||||
- Use Alpine for minimal size
|
||||
- Install only production dependencies
|
||||
- Set NODE_ENV=production
|
||||
|
||||
**Typical:**
|
||||
```dockerfile
|
||||
FROM node:18-alpine AS builder
|
||||
WORKDIR /app
|
||||
COPY package*.json ./
|
||||
RUN npm ci
|
||||
COPY . .
|
||||
RUN npm run build
|
||||
|
||||
FROM node:18-alpine
|
||||
WORKDIR /app
|
||||
COPY --from=builder /app/dist ./dist
|
||||
COPY package*.json ./
|
||||
RUN npm ci --only=production
|
||||
CMD ["node", "dist/index.js"]
|
||||
```
|
||||
|
||||
### Process Management
|
||||
- **PM2**: Production process manager, clustering
|
||||
- **nodemon**: Development auto-reload
|
||||
- **Docker**: Container orchestration handles process management
|
||||
|
||||
### Common Deployment Platforms
|
||||
- **AWS**: Lambda (serverless), ECS/Fargate (containers), EC2, Elastic Beanstalk
|
||||
- **Vercel/Netlify**: Serverless functions, Edge functions
|
||||
- **Heroku**: PaaS, simple deployment
|
||||
- **Google Cloud**: Cloud Run, App Engine, Cloud Functions
|
||||
- **Kubernetes**: Container orchestration
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Code Organization
|
||||
- Separate concerns: routes, controllers, services, models
|
||||
- Use dependency injection (especially with NestJS)
|
||||
- Keep controllers thin, services fat
|
||||
- Use environment variables for configuration
|
||||
|
||||
### Database Integration
|
||||
**Popular ORMs/ODMs:**
|
||||
- **Sequelize**: SQL ORM, supports multiple databases
|
||||
- **TypeORM**: TypeScript-first SQL ORM
|
||||
- **Prisma**: Modern ORM with great DX
|
||||
- **Mongoose**: MongoDB ODM
|
||||
- **Knex**: SQL query builder
|
||||
|
||||
### Error Handling
|
||||
- Use centralized error handler middleware
|
||||
- Create custom error classes
|
||||
- Log errors properly
|
||||
- Return appropriate HTTP status codes
|
||||
|
||||
### Security
|
||||
- Use helmet for security headers
|
||||
- Validate input with joi, yup, or class-validator
|
||||
- Rate limiting with express-rate-limit
|
||||
- CORS configuration
|
||||
- SQL injection prevention via ORM
|
||||
- Don't expose sensitive errors to clients
|
||||
|
||||
### Performance
|
||||
- Use compression middleware
|
||||
- Implement caching (Redis)
|
||||
- Database connection pooling
|
||||
- Optimize queries
|
||||
- Use clustering for CPU-intensive tasks
|
||||
|
||||
### Testing
|
||||
**Frameworks:**
|
||||
- Jest: Most popular, full-featured
|
||||
- Mocha + Chai: Flexible, modular
|
||||
- Supertest: HTTP endpoint testing
|
||||
|
||||
**Patterns:**
|
||||
- Unit tests for services
|
||||
- Integration tests for routes
|
||||
- Mock external dependencies
|
||||
- Use test databases
|
||||
|
||||
### Configuration
|
||||
- dotenv for environment variables
|
||||
- config package for multi-environment configs
|
||||
- Never commit secrets
|
||||
- Use secrets management in production
|
||||
|
||||
### Common Middleware Stack
|
||||
```javascript
|
||||
app.use(helmet()); // Security headers
|
||||
app.use(cors()); // CORS
|
||||
app.use(express.json()); // Parse JSON
|
||||
app.use(compression()); // Compress responses
|
||||
app.use(morgan('combined')); // Logging
|
||||
app.use(rateLimit({...})); // Rate limiting
|
||||
```
|
||||
|
||||
### Monitoring and Observability
|
||||
- **Application monitoring**: New Relic, Datadog, AppDynamics
|
||||
- **Error tracking**: Sentry, Rollbar
|
||||
- **Logging**: Winston, Pino, Bunyan
|
||||
- **APM**: OpenTelemetry, AWS X-Ray
|
||||
|
||||
### Package Management
|
||||
- Use npm or yarn consistently
|
||||
- Lock dependencies with package-lock.json or yarn.lock
|
||||
- Regular dependency updates
|
||||
- Check for vulnerabilities: npm audit
|
||||
|
||||
## Technology-Specific Patterns
|
||||
|
||||
### Real-time Applications
|
||||
- Socket.io for WebSockets
|
||||
- Server-Sent Events (SSE) for one-way updates
|
||||
- Redis pub/sub for scaling WebSockets
|
||||
|
||||
### GraphQL APIs
|
||||
- Apollo Server
|
||||
- Type-GraphQL with TypeScript
|
||||
- DataLoader for batching and caching
|
||||
|
||||
### Background Jobs
|
||||
- Bull (Redis-based queue)
|
||||
- Agenda (MongoDB-based)
|
||||
- BeeQueue (lightweight)
|
||||
- AWS SQS for serverless
|
||||
|
||||
### API Documentation
|
||||
- Swagger/OpenAPI with swagger-jsdoc
|
||||
- API Blueprint
|
||||
- Postman collections
|
||||
383
skills/references/python.md
Normal file
383
skills/references/python.md
Normal file
@@ -0,0 +1,383 @@
|
||||
# Python Architecture Patterns
|
||||
|
||||
## Table of Contents
|
||||
- Common Frameworks and Patterns
|
||||
- Project Structure Conventions
|
||||
- Deployment Considerations
|
||||
- Best Practices
|
||||
|
||||
## Common Frameworks and Patterns
|
||||
|
||||
### Django
|
||||
**Typical Structure:**
|
||||
```
|
||||
project/
|
||||
├── manage.py
|
||||
├── project_name/ # Project config
|
||||
│ ├── settings.py
|
||||
│ ├── urls.py
|
||||
│ └── wsgi.py
|
||||
├── app_name/ # Django apps
|
||||
│ ├── models.py
|
||||
│ ├── views.py
|
||||
│ ├── serializers.py # If using DRF
|
||||
│ ├── urls.py
|
||||
│ └── admin.py
|
||||
└── requirements.txt
|
||||
```
|
||||
|
||||
**Key Characteristics:**
|
||||
- "Batteries included" framework
|
||||
- ORM built-in
|
||||
- Admin interface out of the box
|
||||
- Best for full-stack web applications
|
||||
- Django REST Framework for APIs
|
||||
|
||||
### Flask
|
||||
**Typical Structure:**
|
||||
```
|
||||
project/
|
||||
├── app/
|
||||
│ ├── __init__.py # Application factory
|
||||
│ ├── routes/
|
||||
│ ├── models/
|
||||
│ ├── services/
|
||||
│ └── utils/
|
||||
├── tests/
|
||||
├── config.py
|
||||
├── requirements.txt
|
||||
└── run.py
|
||||
```
|
||||
|
||||
**Key Characteristics:**
|
||||
- Microframework, minimal
|
||||
- Flexible and unopinionated
|
||||
- Extension-based (Flask-SQLAlchemy, Flask-JWT, etc.)
|
||||
- Good for APIs and smaller applications
|
||||
|
||||
### FastAPI
|
||||
**Typical Structure:**
|
||||
```
|
||||
project/
|
||||
├── app/
|
||||
│ ├── main.py
|
||||
│ ├── routers/
|
||||
│ ├── models/
|
||||
│ ├── schemas/ # Pydantic models
|
||||
│ ├── services/
|
||||
│ └── dependencies.py
|
||||
├── tests/
|
||||
└── requirements.txt
|
||||
```
|
||||
|
||||
**Key Characteristics:**
|
||||
- Modern, high-performance
|
||||
- Type hints and validation (Pydantic)
|
||||
- Automatic API documentation (Swagger/ReDoc)
|
||||
- Async support built-in
|
||||
- Ideal for APIs and microservices
|
||||
|
||||
## Project Structure Conventions
|
||||
|
||||
### API Application (Flask/FastAPI)
|
||||
```
|
||||
project/
|
||||
├── app/
|
||||
│ ├── api/
|
||||
│ │ ├── v1/
|
||||
│ │ │ ├── endpoints/
|
||||
│ │ │ └── dependencies.py
|
||||
│ │ └── v2/
|
||||
│ ├── core/
|
||||
│ │ ├── config.py
|
||||
│ │ └── security.py
|
||||
│ ├── models/
|
||||
│ ├── schemas/
|
||||
│ ├── services/
|
||||
│ └── db/
|
||||
├── tests/
|
||||
├── alembic/ # Database migrations
|
||||
├── .env
|
||||
└── requirements.txt
|
||||
```
|
||||
|
||||
### Django Monolith
|
||||
```
|
||||
project/
|
||||
├── manage.py
|
||||
├── project_name/
|
||||
│ ├── settings/
|
||||
│ │ ├── base.py
|
||||
│ │ ├── dev.py
|
||||
│ │ └── prod.py
|
||||
│ ├── urls.py
|
||||
│ └── wsgi.py
|
||||
├── apps/
|
||||
│ ├── users/
|
||||
│ ├── orders/
|
||||
│ └── products/
|
||||
├── static/
|
||||
├── templates/
|
||||
└── requirements/
|
||||
```
|
||||
|
||||
### Data Science/ML Application
|
||||
```
|
||||
project/
|
||||
├── data/
|
||||
│ ├── raw/
|
||||
│ ├── processed/
|
||||
│ └── models/
|
||||
├── notebooks/
|
||||
├── src/
|
||||
│ ├── data/
|
||||
│ ├── features/
|
||||
│ ├── models/
|
||||
│ └── visualization/
|
||||
├── tests/
|
||||
└── requirements.txt
|
||||
```
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
### WSGI/ASGI Servers
|
||||
**Production servers:**
|
||||
- **Gunicorn**: WSGI server for Django/Flask
|
||||
- **Uvicorn**: ASGI server for FastAPI
|
||||
- **uWSGI**: Full-featured application server
|
||||
- **Hypercorn**: ASGI server alternative
|
||||
|
||||
**Typical setup:**
|
||||
```bash
|
||||
gunicorn -w 4 -b 0.0.0.0:8000 app:app # Flask
|
||||
gunicorn -w 4 myproject.wsgi:application # Django
|
||||
uvicorn app.main:app --workers 4 # FastAPI
|
||||
```
|
||||
|
||||
### Containerization
|
||||
**Dockerfile patterns:**
|
||||
```dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy application
|
||||
COPY . .
|
||||
|
||||
# Run as non-root user
|
||||
RUN useradd -m appuser && chown -R appuser /app
|
||||
USER appuser
|
||||
|
||||
CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:8000", "app:app"]
|
||||
```
|
||||
|
||||
### Common Deployment Platforms
|
||||
- **AWS**: Lambda (Zappa, Chalice), ECS, EC2, Elastic Beanstalk
|
||||
- **Google Cloud**: Cloud Run, App Engine, Compute Engine
|
||||
- **Azure**: App Service, Functions, Container Instances
|
||||
- **Heroku**: Simple deployment with Procfile
|
||||
- **PythonAnywhere**: Python-specific hosting
|
||||
- **Kubernetes**: Container orchestration
|
||||
|
||||
### Virtual Environments
|
||||
- **venv**: Built-in Python 3.3+
|
||||
- **virtualenv**: Cross-version compatibility
|
||||
- **conda**: Data science focused
|
||||
- **Poetry**: Modern dependency management
|
||||
- **pipenv**: Combines pip and virtualenv
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Code Organization
|
||||
- Follow PEP 8 style guide
|
||||
- Use blueprints (Flask) or apps (Django) for modularity
|
||||
- Separate business logic from views/routes
|
||||
- Keep models lean, use services for complex logic
|
||||
|
||||
### Database Integration
|
||||
|
||||
**Django ORM:**
|
||||
- Built-in, powerful ORM
|
||||
- Migrations with django-admin
|
||||
- QuerySet API
|
||||
|
||||
**SQLAlchemy (Flask/FastAPI):**
|
||||
- Most popular ORM
|
||||
- Core and ORM layers
|
||||
- Alembic for migrations
|
||||
|
||||
**Alternatives:**
|
||||
- **Tortoise ORM**: AsyncIO ORM for FastAPI
|
||||
- **Peewee**: Lightweight ORM
|
||||
- **asyncpg/psycopg3**: Direct database drivers
|
||||
|
||||
### Configuration Management
|
||||
```python
|
||||
# Use environment-specific configs
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
class Config:
|
||||
DEBUG = os.getenv('DEBUG', False)
|
||||
DATABASE_URL = os.getenv('DATABASE_URL')
|
||||
SECRET_KEY = os.getenv('SECRET_KEY')
|
||||
|
||||
class DevelopmentConfig(Config):
|
||||
DEBUG = True
|
||||
|
||||
class ProductionConfig(Config):
|
||||
DEBUG = False
|
||||
```
|
||||
|
||||
### Security
|
||||
|
||||
**Django:**
|
||||
- CSRF protection built-in
|
||||
- XSS protection
|
||||
- SQL injection prevention via ORM
|
||||
- Security middleware enabled
|
||||
|
||||
**Flask/FastAPI:**
|
||||
- Use Flask-Security or FastAPI security utilities
|
||||
- Validate input with Pydantic (FastAPI) or marshmallow (Flask)
|
||||
- CORS: flask-cors or FastAPI CORSMiddleware
|
||||
- Rate limiting: Flask-Limiter
|
||||
|
||||
**General:**
|
||||
- Never commit secrets
|
||||
- Use environment variables
|
||||
- Validate and sanitize input
|
||||
- Use parameterized queries
|
||||
- Keep dependencies updated
|
||||
|
||||
### Error Handling
|
||||
```python
|
||||
# Centralized error handling
|
||||
from fastapi import HTTPException
|
||||
from flask import jsonify
|
||||
|
||||
@app.errorhandler(404)
|
||||
def not_found(error):
|
||||
return jsonify({"error": "Not found"}), 404
|
||||
|
||||
@app.exception_handler(ValueError)
|
||||
async def value_error_handler(request, exc):
|
||||
return JSONResponse(
|
||||
status_code=400,
|
||||
content={"message": str(exc)}
|
||||
)
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
**Frameworks:**
|
||||
- **pytest**: Most popular, full-featured
|
||||
- **unittest**: Built-in
|
||||
- **nose2**: Extension of unittest
|
||||
|
||||
**Tools:**
|
||||
- **pytest-cov**: Coverage reporting
|
||||
- **factory_boy**: Test fixtures
|
||||
- **faker**: Generate fake data
|
||||
- **responses/httpx**: Mock HTTP requests
|
||||
|
||||
**Patterns:**
|
||||
- Unit tests for business logic
|
||||
- Integration tests for endpoints
|
||||
- Use fixtures for test data
|
||||
- Mock external services
|
||||
|
||||
### Logging
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
```
|
||||
|
||||
**Production logging:**
|
||||
- Use structured logging (structlog)
|
||||
- Log to stdout/stderr for containers
|
||||
- Integrate with ELK, Datadog, CloudWatch
|
||||
|
||||
### Package Management
|
||||
```txt
|
||||
# requirements.txt
|
||||
Django==4.2.0
|
||||
psycopg2-binary==2.9.6
|
||||
gunicorn==20.1.0
|
||||
|
||||
# Or use Poetry
|
||||
# pyproject.toml
|
||||
[tool.poetry.dependencies]
|
||||
python = "^3.11"
|
||||
django = "^4.2"
|
||||
```
|
||||
|
||||
### Type Hints
|
||||
```python
|
||||
from typing import List, Optional
|
||||
|
||||
def get_users(limit: int = 10) -> List[User]:
|
||||
return User.objects.all()[:limit]
|
||||
|
||||
# Use mypy for static type checking
|
||||
```
|
||||
|
||||
### Async/Await (FastAPI, modern Python)
|
||||
```python
|
||||
async def get_data():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get('https://api.example.com')
|
||||
return response.json()
|
||||
```
|
||||
|
||||
## Technology-Specific Patterns
|
||||
|
||||
### REST APIs
|
||||
- Django REST Framework: Full-featured, batteries included
|
||||
- Flask-RESTX: Flask extension for APIs
|
||||
- FastAPI: Modern, automatic docs, high performance
|
||||
|
||||
### Background Jobs
|
||||
- **Celery**: Distributed task queue (with Redis/RabbitMQ)
|
||||
- **RQ**: Simple Redis-based queue
|
||||
- **Dramatiq**: Alternative to Celery
|
||||
- **APScheduler**: Scheduler for periodic tasks
|
||||
|
||||
### Caching
|
||||
- **Redis**: Most common
|
||||
- **Memcached**: Simple key-value store
|
||||
- **Django cache framework**: Built-in abstraction
|
||||
- **Flask-Caching**: Flask extension
|
||||
|
||||
### GraphQL
|
||||
- **Graphene-Django**: Django integration
|
||||
- **Strawberry**: Modern, type-hint based
|
||||
- **Ariadne**: Schema-first approach
|
||||
|
||||
### WebSockets
|
||||
- **Django Channels**: Async Django
|
||||
- **Flask-SocketIO**: WebSocket support for Flask
|
||||
- **FastAPI WebSockets**: Built-in support
|
||||
|
||||
### API Documentation
|
||||
- FastAPI: Auto-generated (Swagger/ReDoc)
|
||||
- Django REST Framework: Browsable API + schema generation
|
||||
- Flask: flask-swagger or flasgger
|
||||
|
||||
### Monitoring
|
||||
- **Sentry**: Error tracking
|
||||
- **New Relic/Datadog**: APM
|
||||
- **Prometheus**: Metrics
|
||||
- **django-debug-toolbar**: Development profiling
|
||||
390
skills/references/serverless.md
Normal file
390
skills/references/serverless.md
Normal file
@@ -0,0 +1,390 @@
|
||||
# Serverless Architecture Patterns
|
||||
|
||||
## Table of Contents
|
||||
- Core Concepts
|
||||
- Function Patterns
|
||||
- Data Patterns
|
||||
- Integration Patterns
|
||||
- Best Practices
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Function as a Service (FaaS)
|
||||
Code runs in response to events without managing servers.
|
||||
|
||||
**Key Characteristics:**
|
||||
- Event-driven execution
|
||||
- Automatic scaling
|
||||
- Pay-per-execution pricing
|
||||
- Stateless functions
|
||||
- Short-lived execution
|
||||
|
||||
**Major Providers:**
|
||||
- AWS Lambda
|
||||
- Google Cloud Functions
|
||||
- Azure Functions
|
||||
- Cloudflare Workers
|
||||
|
||||
### Backend as a Service (BaaS)
|
||||
Managed services replace traditional backend components.
|
||||
|
||||
**Examples:**
|
||||
- Database: DynamoDB, Firestore, Fauna
|
||||
- Auth: Auth0, AWS Cognito, Firebase Auth
|
||||
- Storage: S3, Cloud Storage
|
||||
- APIs: API Gateway, AppSync
|
||||
|
||||
## Function Patterns
|
||||
|
||||
### API Endpoint Pattern
|
||||
```
|
||||
API Gateway --trigger--> Lambda Function --response--> API Gateway
|
||||
```
|
||||
|
||||
**Use case:** REST API, GraphQL API
|
||||
|
||||
**Example (AWS):**
|
||||
```
|
||||
API Gateway (POST /users)
|
||||
↓
|
||||
Lambda (createUser)
|
||||
↓
|
||||
DynamoDB
|
||||
```
|
||||
|
||||
### Event Processing Pattern
|
||||
```
|
||||
Event Source --event--> Lambda --process--> Output
|
||||
```
|
||||
|
||||
**Event Sources:**
|
||||
- S3: File uploads
|
||||
- DynamoDB Streams: Database changes
|
||||
- SQS: Message queue
|
||||
- SNS: Notifications
|
||||
- EventBridge: Custom events
|
||||
|
||||
**Use case:** Image processing, data transformation, ETL
|
||||
|
||||
### Scheduled Jobs Pattern
|
||||
```
|
||||
CloudWatch Events (cron) --trigger--> Lambda
|
||||
```
|
||||
|
||||
**Use case:** Batch processing, cleanup jobs, reports
|
||||
|
||||
**Example:**
|
||||
```
|
||||
EventBridge Rule (rate(1 day))
|
||||
↓
|
||||
Lambda (dailyReport)
|
||||
↓
|
||||
Send Email / Store Results
|
||||
```
|
||||
|
||||
### Stream Processing Pattern
|
||||
```
|
||||
Kinesis Stream --records--> Lambda --aggregate/transform--> Destination
|
||||
```
|
||||
|
||||
**Use case:** Real-time analytics, log processing
|
||||
|
||||
## Data Patterns
|
||||
|
||||
### DynamoDB Pattern (NoSQL)
|
||||
|
||||
**Access Patterns:**
|
||||
- Single table design
|
||||
- GSI for query flexibility
|
||||
- DynamoDB Streams for CDC
|
||||
|
||||
**Best practices:**
|
||||
- Design for access patterns first
|
||||
- Use partition keys for distribution
|
||||
- Batch operations when possible
|
||||
- Enable auto-scaling
|
||||
|
||||
### Aurora Serverless Pattern (SQL)
|
||||
|
||||
**Features:**
|
||||
- Auto-scaling SQL database
|
||||
- Pay per use
|
||||
- Data API for HTTP access
|
||||
|
||||
**Use case:** Variable workloads, development environments
|
||||
|
||||
### Multi-Database Pattern
|
||||
|
||||
**Polyglot persistence:**
|
||||
- DynamoDB for user profiles
|
||||
- S3 for files
|
||||
- ElastiCache for sessions
|
||||
- RDS for relational data
|
||||
|
||||
### API Composition Pattern
|
||||
|
||||
Function aggregates data from multiple sources:
|
||||
```
|
||||
Lambda
|
||||
├── Call Service A API
|
||||
├── Call Service B API
|
||||
├── Query DynamoDB
|
||||
└── Aggregate results
|
||||
```
|
||||
|
||||
## Integration Patterns
|
||||
|
||||
### API Gateway Integration
|
||||
|
||||
**Types:**
|
||||
- Lambda Proxy: Full HTTP request to function
|
||||
- Lambda Integration: Custom request/response mapping
|
||||
- HTTP Proxy: Direct passthrough to backend
|
||||
- Mock: Return static response
|
||||
|
||||
**Features:**
|
||||
- Request validation
|
||||
- Authorization
|
||||
- Rate limiting
|
||||
- Caching
|
||||
- CORS
|
||||
|
||||
### EventBridge Pattern
|
||||
|
||||
Central event bus for service integration:
|
||||
```
|
||||
Service A --event--> EventBridge --route--> Lambda B
|
||||
--route--> Lambda C
|
||||
--route--> SQS Queue
|
||||
```
|
||||
|
||||
**Use case:** Event-driven architecture, decoupling services
|
||||
|
||||
### Step Functions Pattern
|
||||
|
||||
Orchestrate multiple Lambda functions:
|
||||
```
|
||||
Step Functions State Machine
|
||||
↓
|
||||
Step 1: Validate (Lambda)
|
||||
↓
|
||||
Step 2: Process (Lambda)
|
||||
↓
|
||||
Step 3: Notify (Lambda)
|
||||
```
|
||||
|
||||
**Use case:** Workflows, long-running processes, error handling
|
||||
|
||||
### SQS Queue Pattern
|
||||
|
||||
Decouple and buffer workloads:
|
||||
```
|
||||
Producer --message--> SQS Queue --poll--> Lambda Consumer
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- Async processing
|
||||
- Rate limiting
|
||||
- Retry logic
|
||||
- Dead letter queue
|
||||
|
||||
### Fan-out Pattern
|
||||
|
||||
Single event triggers multiple functions:
|
||||
```
|
||||
S3 Upload Event
|
||||
↓
|
||||
SNS Topic
|
||||
├── Lambda: Create Thumbnail
|
||||
├── Lambda: Extract Metadata
|
||||
└── Lambda: Virus Scan
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Cold Start Optimization
|
||||
|
||||
**Strategies:**
|
||||
- Keep functions small
|
||||
- Minimize dependencies
|
||||
- Use provisioned concurrency (critical paths)
|
||||
- Reuse connections
|
||||
- Initialize outside handler
|
||||
|
||||
```javascript
|
||||
// Good: Initialize outside
|
||||
const db = new DynamoDB.DocumentClient();
|
||||
|
||||
exports.handler = async (event) => {
|
||||
// Use db here
|
||||
};
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
**Patterns:**
|
||||
- Return errors explicitly
|
||||
- Use Dead Letter Queues
|
||||
- Implement retry logic
|
||||
- Log errors with context
|
||||
|
||||
```javascript
|
||||
exports.handler = async (event) => {
|
||||
try {
|
||||
// Process event
|
||||
} catch (error) {
|
||||
console.error('Error:', error);
|
||||
// Send to DLQ or SNS
|
||||
throw error; // Trigger retry
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### Security
|
||||
|
||||
**Best practices:**
|
||||
- Least privilege IAM roles
|
||||
- Encrypt environment variables
|
||||
- Use VPC for private resources
|
||||
- Validate input
|
||||
- Use AWS Secrets Manager
|
||||
|
||||
### Cost Optimization
|
||||
|
||||
**Strategies:**
|
||||
- Right-size memory (affects CPU)
|
||||
- Optimize execution time
|
||||
- Use SQS for buffering
|
||||
- Set appropriate timeouts
|
||||
- Monitor unused functions
|
||||
|
||||
### Observability
|
||||
|
||||
**Logging:**
|
||||
- Use structured logging (JSON)
|
||||
- Include correlation IDs
|
||||
- Log key metrics
|
||||
|
||||
**Monitoring:**
|
||||
- CloudWatch Metrics
|
||||
- X-Ray for tracing
|
||||
- Custom metrics
|
||||
|
||||
**Alerts:**
|
||||
- Error rates
|
||||
- Duration
|
||||
- Throttles
|
||||
- Dead letter queue depth
|
||||
|
||||
### State Management
|
||||
|
||||
**Stateless functions:**
|
||||
- No local state
|
||||
- Use external stores (DynamoDB, S3, ElastiCache)
|
||||
- Pass state in events
|
||||
|
||||
**Step Functions for stateful workflows:**
|
||||
- Maintain workflow state
|
||||
- Handle long-running processes
|
||||
|
||||
### Testing
|
||||
|
||||
**Levels:**
|
||||
- Unit: Test handler logic
|
||||
- Integration: Test with local services (LocalStack)
|
||||
- E2E: Test in AWS
|
||||
|
||||
**Tools:**
|
||||
- Jest/Mocha for unit tests
|
||||
- AWS SAM for local testing
|
||||
- LocalStack for local AWS
|
||||
|
||||
### Deployment
|
||||
|
||||
**IaC Tools:**
|
||||
- AWS SAM (Serverless Application Model)
|
||||
- Serverless Framework
|
||||
- AWS CDK
|
||||
- Terraform
|
||||
|
||||
**CI/CD:**
|
||||
- Automated testing
|
||||
- Canary deployments
|
||||
- Blue-green deployments
|
||||
- Rollback capability
|
||||
|
||||
## Architecture Examples
|
||||
|
||||
### REST API
|
||||
```
|
||||
API Gateway (REST)
|
||||
├── GET /items --> Lambda (listItems) --> DynamoDB
|
||||
├── POST /items --> Lambda (createItem) --> DynamoDB
|
||||
└── GET /items/{id} --> Lambda (getItem) --> DynamoDB
|
||||
```
|
||||
|
||||
### Event-Driven Processing
|
||||
```
|
||||
S3 Bucket (image upload)
|
||||
↓
|
||||
Lambda (processImage)
|
||||
├── Resize image
|
||||
├── Store in S3
|
||||
└── Update DynamoDB
|
||||
```
|
||||
|
||||
### Microservices
|
||||
```
|
||||
Service 1: API Gateway + Lambda + DynamoDB
|
||||
Service 2: API Gateway + Lambda + RDS Aurora
|
||||
Service 3: EventBridge + Lambda + SQS
|
||||
|
||||
Inter-service: HTTP APIs or EventBridge
|
||||
```
|
||||
|
||||
### Data Pipeline
|
||||
```
|
||||
S3 (raw data)
|
||||
↓
|
||||
Lambda (transform)
|
||||
↓
|
||||
Kinesis Firehose
|
||||
↓
|
||||
S3 (processed data)
|
||||
↓
|
||||
Athena (query)
|
||||
```
|
||||
|
||||
## Common Challenges
|
||||
|
||||
### Vendor Lock-in
|
||||
**Mitigation:** Use frameworks (Serverless Framework), abstract provider-specific code
|
||||
|
||||
### Cold Starts
|
||||
**Mitigation:** Provisioned concurrency, keep functions warm, optimize bundle size
|
||||
|
||||
### Debugging
|
||||
**Mitigation:** Structured logging, X-Ray tracing, local testing
|
||||
|
||||
### Testing
|
||||
**Mitigation:** Unit tests, integration tests with LocalStack, E2E tests in staging
|
||||
|
||||
### Distributed Tracing
|
||||
**Mitigation:** AWS X-Ray, correlation IDs, structured logs
|
||||
|
||||
## When to Use Serverless
|
||||
|
||||
**Good for:**
|
||||
- Variable workload
|
||||
- Event-driven applications
|
||||
- APIs with unpredictable traffic
|
||||
- Scheduled jobs
|
||||
- Rapid prototyping
|
||||
- Startups (reduce ops overhead)
|
||||
|
||||
**Not ideal for:**
|
||||
- Long-running processes (>15 min)
|
||||
- Consistent high-throughput
|
||||
- Applications requiring low latency
|
||||
- Complex state management
|
||||
- Heavy computation (cost)
|
||||
156
skills/references/workflows.md
Normal file
156
skills/references/workflows.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Architecture Documentation Workflows
|
||||
|
||||
## Table of Contents
|
||||
- Interview Workflow for New Projects
|
||||
- Interview Workflow for Existing Systems
|
||||
- Documentation Generation Process
|
||||
- Update and Maintenance Workflow
|
||||
|
||||
## Interview Workflow for New Projects
|
||||
|
||||
Gather information through these questions in order:
|
||||
|
||||
### Phase 1: Project Context (2-3 questions)
|
||||
1. **Project basics**: Name, purpose, target users
|
||||
2. **Timeline**: Launch date, current phase (planning/development/production)
|
||||
3. **Team**: Size, composition, experience level
|
||||
|
||||
### Phase 2: Technical Stack (2-4 questions)
|
||||
1. **Backend**: Primary language and framework (e.g., "Node.js with Express" or "Python with Django")
|
||||
2. **Frontend**: Framework or approach (e.g., "React SPA" or "Server-rendered with templates")
|
||||
3. **Database**: Type and name (e.g., "PostgreSQL" or "MongoDB")
|
||||
4. **Infrastructure**: Cloud provider or hosting (e.g., "AWS" or "On-premise")
|
||||
|
||||
### Phase 3: Architecture Pattern (1-2 questions)
|
||||
1. **Pattern**: Monolith, microservices, serverless, or hybrid
|
||||
2. **Rationale**: Why this pattern? (helps understand constraints)
|
||||
|
||||
### Phase 4: Components (2-4 questions based on pattern)
|
||||
|
||||
For Monolith:
|
||||
- Main modules or features
|
||||
- Background jobs or scheduled tasks
|
||||
- Static asset handling
|
||||
|
||||
For Microservices:
|
||||
- List of services (name and responsibility)
|
||||
- How services communicate (REST, gRPC, message queue)
|
||||
- Shared components (API gateway, service mesh)
|
||||
|
||||
For Serverless:
|
||||
- Functions and their triggers
|
||||
- State management approach
|
||||
- Event sources
|
||||
|
||||
### Phase 5: Data and Integrations (1-3 questions)
|
||||
1. **Additional data stores**: Cache, search engine, file storage
|
||||
2. **External services**: Payment, email, analytics, etc.
|
||||
3. **APIs**: Third-party APIs being consumed
|
||||
|
||||
### Phase 6: Operations (1-2 questions)
|
||||
1. **Deployment**: Manual, CI/CD, orchestration
|
||||
2. **Monitoring**: Logging, metrics, alerting setup
|
||||
|
||||
## Interview Workflow for Existing Systems
|
||||
|
||||
For systems already built, adjust approach:
|
||||
|
||||
### Quick Assessment
|
||||
1. **Documentation exists?** If yes, review it first
|
||||
2. **Codebase access?** If yes, analyze structure
|
||||
3. **Team knowledge?** Interview developers
|
||||
|
||||
### Targeted Questions
|
||||
1. What's changed since last documentation?
|
||||
2. What's causing confusion for new developers?
|
||||
3. What's undocumented or poorly documented?
|
||||
4. Are there any planned changes?
|
||||
|
||||
### Validation Approach
|
||||
- Cross-reference answers with code
|
||||
- Identify discrepancies between docs and reality
|
||||
- Highlight areas needing attention
|
||||
|
||||
## Documentation Generation Process
|
||||
|
||||
### Step 1: Select Template
|
||||
Choose based on architecture pattern:
|
||||
- `assets/ARCHITECTURE.md` for general/unknown
|
||||
- `assets/ARCHITECTURE-microservices.md` for microservices
|
||||
- `assets/ARCHITECTURE-monolith.md` for monoliths
|
||||
|
||||
### Step 2: Generate System Diagram
|
||||
Use `scripts/generate_diagram.py` based on pattern:
|
||||
- Monolith: Use "layered" type
|
||||
- Microservices: Use "flow" or "c4" type
|
||||
- Simple systems: Use "simple" type
|
||||
|
||||
### Step 3: Fill Template Sections
|
||||
Populate in this order:
|
||||
1. Project Identification (Section 10) - establishes context
|
||||
2. Project Structure (Section 1) - concrete foundation
|
||||
3. System Diagram (Section 2) - using generated diagram
|
||||
4. Core Components (Section 3) - from interview data
|
||||
5. Data Stores (Section 4) - databases and caches
|
||||
6. External Integrations (Section 5) - third-party services
|
||||
7. Deployment & Infrastructure (Section 6) - ops details
|
||||
8. Security Considerations (Section 7) - auth, encryption
|
||||
9. Development & Testing (Section 8) - dev environment
|
||||
10. Future Considerations (Section 9) - roadmap items
|
||||
11. Glossary (Section 11) - domain-specific terms
|
||||
|
||||
### Step 4: Enhance with Technology Specifics
|
||||
Load relevant reference and apply patterns:
|
||||
- Node.js: See references/nodejs.md
|
||||
- Python: See references/python.md
|
||||
- Java: See references/java.md
|
||||
|
||||
### Step 5: Validate
|
||||
Run validation script:
|
||||
```bash
|
||||
python scripts/validate_architecture.py ARCHITECTURE.md
|
||||
```
|
||||
|
||||
Address any issues or warnings.
|
||||
|
||||
## Update and Maintenance Workflow
|
||||
|
||||
### For Incremental Updates
|
||||
1. Identify changed components
|
||||
2. Update affected sections only
|
||||
3. Update "Date of Last Update" in Section 10
|
||||
4. Add brief note in Future Considerations if major change
|
||||
|
||||
### For Major Updates
|
||||
1. Re-interview on changed areas
|
||||
2. Regenerate system diagram if structure changed
|
||||
3. Update multiple sections as needed
|
||||
4. Consider adding version notes
|
||||
|
||||
### Best Practices
|
||||
- Keep updates small and frequent
|
||||
- Document changes when they happen
|
||||
- Review quarterly even if no changes
|
||||
- Archive old versions if major restructure
|
||||
|
||||
## Tips for Effective Interviews
|
||||
|
||||
### Keep Questions Focused
|
||||
- Ask one thing at a time
|
||||
- Avoid overwhelming with options
|
||||
- Build on previous answers
|
||||
|
||||
### Adapt Based on Responses
|
||||
- If user is technical, use technical terms
|
||||
- If user is non-technical, simplify language
|
||||
- Skip redundant questions
|
||||
|
||||
### Handle Uncertainty
|
||||
- If user unsure, offer to add placeholder
|
||||
- Suggest reasonable defaults
|
||||
- Mark uncertain items for review
|
||||
|
||||
### Validate Understanding
|
||||
- Summarize what you heard
|
||||
- Confirm critical details
|
||||
- Check for consistency
|
||||
Reference in New Issue
Block a user