zhongwei/gh-hermeticormus-alqvimia-contador-plugins-developer-essentials
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Zhongwei Li
2025-11-29 18:34:03 +08:00
commit 0a77775e5d
11 changed files with 4474 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
.claude-plugin/plugin.json Normal file
View File

@@ -0,0 +1,19 @@
{
"name": "developer-essentials",
"description": "Essential developer skills including Git workflows, SQL optimization, error handling, code review, E2E testing, authentication, debugging, and monorepo management",
"version": "1.0.0",
"author": {
"name": "Seth Hobson",
"url": "https://github.com/wshobson"
},
"skills": [
"./skills/git-advanced-workflows",
"./skills/sql-optimization-patterns",
"./skills/error-handling-patterns",
"./skills/code-review-excellence",
"./skills/e2e-testing-patterns",
"./skills/auth-implementation-patterns",
"./skills/debugging-strategies",
"./skills/monorepo-management"
]
}

3
README.md Normal file
View File

@@ -0,0 +1,3 @@
# developer-essentials
Essential developer skills including Git workflows, SQL optimization, error handling, code review, E2E testing, authentication, debugging, and monorepo management

73
plugin.lock.json Normal file
View File

@@ -0,0 +1,73 @@
{
"$schema": "internal://schemas/plugin.lock.v1.json",
"pluginId": "gh:HermeticOrmus/Alqvimia-Contador:plugins/developer-essentials",
"normalized": {
"repo": null,
"ref": "refs/tags/v20251128.0",
"commit": "18b7e4d59f9d3b51b7b63d95ca66913c5f1b5a56",
"treeHash": "c9b7edcd30393edaaed9e09cd5a86a010d517de4cb91a58695543cc9cf83ebcc",
"generatedAt": "2025-11-28T10:10:45.722320Z",
"toolVersion": "publish_plugins.py@0.2.0"
},
"origin": {
"remote": "git@github.com:zhongweili/42plugin-data.git",
"branch": "master",
"commit": "aa1497ed0949fd50e99e70d6324a29c5b34f9390",
"repoRoot": "/Users/zhongweili/projects/openmind/42plugin-data"
},
"manifest": {
"name": "developer-essentials",
"description": "Essential developer skills including Git workflows, SQL optimization, error handling, code review, E2E testing, authentication, debugging, and monorepo management",
"version": "1.0.0"
},
"content": {
"files": [
{
"path": "README.md",
"sha256": "736fdacef932bb4dda27b3bb6be04af6f3e02ca8741c7a893933d014023ba889"
},
{
"path": ".claude-plugin/plugin.json",
"sha256": "62aabbf4f6e43431c56554fd009871507f6cfa2ce1bbcc689af46467216cabf4"
},
{
"path": "skills/auth-implementation-patterns/SKILL.md",
"sha256": "bbc8c8de4a53d0c99146bb56492b0842a7f049d234cf5628481bc16e5637901b"
},
{
"path": "skills/monorepo-management/SKILL.md",
"sha256": "5e5e9486a348a397c0d4ef8e7bb4d858ab74b65ba7901eca0694efd61d90fafc"
},
{
"path": "skills/error-handling-patterns/SKILL.md",
"sha256": "aa8f11361c893893a0007b58f9322bb92cfcffa5b7b91f506059935192a3e68d"
},
{
"path": "skills/debugging-strategies/SKILL.md",
"sha256": "61dbd539f0f9ba90aec588147d7ee44a9c54a4dcf148011aea9f000b2cd724a7"
},
{
"path": "skills/code-review-excellence/SKILL.md",
"sha256": "235ec26133c43da32516fb2fbc6000e3dad78beb7e252b35577bf51346e1128d"
},
{
"path": "skills/sql-optimization-patterns/SKILL.md",
"sha256": "bfd74fb05aa7e6cc40358f6360043f1db9c2c914b5dd9a22bd82b43d49b86a8a"
},
{
"path": "skills/e2e-testing-patterns/SKILL.md",
"sha256": "c2efe16ab57630f3f04a580173661dd9ddb2c328ff5724c6d394a153c22ca52c"
},
{
"path": "skills/git-advanced-workflows/SKILL.md",
"sha256": "624ae93393ccc85ff59fd1c02e8da853ded75ed378c53f0115dc6c71f37163db"
}
],
"dirSha256": "c9b7edcd30393edaaed9e09cd5a86a010d517de4cb91a58695543cc9cf83ebcc"
},
"security": {
"scannedAt": null,
"scannerVersion": null,
"flags": []
}
}

634
skills/auth-implementation-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,634 @@
---
name: auth-implementation-patterns
description: Master authentication and authorization patterns including JWT, OAuth2, session management, and RBAC to build secure, scalable access control systems. Use when implementing auth systems, securing APIs, or debugging security issues.
---
# Authentication & Authorization Implementation Patterns
Build secure, scalable authentication and authorization systems using industry-standard patterns and modern best practices.
## When to Use This Skill
- Implementing user authentication systems
- Securing REST or GraphQL APIs
- Adding OAuth2/social login
- Implementing role-based access control (RBAC)
- Designing session management
- Migrating authentication systems
- Debugging auth issues
- Implementing SSO or multi-tenancy
## Core Concepts
### 1. Authentication vs Authorization
**Authentication (AuthN)**: Who are you?
- Verifying identity (username/password, OAuth, biometrics)
- Issuing credentials (sessions, tokens)
- Managing login/logout
**Authorization (AuthZ)**: What can you do?
- Permission checking
- Role-based access control (RBAC)
- Resource ownership validation
- Policy enforcement
### 2. Authentication Strategies
**Session-Based:**
- Server stores session state
- Session ID in cookie
- Traditional, simple, stateful
**Token-Based (JWT):**
- Stateless, self-contained
- Scales horizontally
- Can store claims
**OAuth2/OpenID Connect:**
- Delegate authentication
- Social login (Google, GitHub)
- Enterprise SSO
## JWT Authentication
### Pattern 1: JWT Implementation
```typescript
// JWT structure: header.payload.signature
import jwt from 'jsonwebtoken';
import { Request, Response, NextFunction } from 'express';
interface JWTPayload {
userId: string;
email: string;
role: string;
iat: number;
exp: number;
}
// Generate JWT
function generateTokens(userId: string, email: string, role: string) {
const accessToken = jwt.sign(
{ userId, email, role },
process.env.JWT_SECRET!,
{ expiresIn: '15m' } // Short-lived
);
const refreshToken = jwt.sign(
{ userId },
process.env.JWT_REFRESH_SECRET!,
{ expiresIn: '7d' } // Long-lived
);
return { accessToken, refreshToken };
}
// Verify JWT
function verifyToken(token: string): JWTPayload {
try {
return jwt.verify(token, process.env.JWT_SECRET!) as JWTPayload;
} catch (error) {
if (error instanceof jwt.TokenExpiredError) {
throw new Error('Token expired');
}
if (error instanceof jwt.JsonWebTokenError) {
throw new Error('Invalid token');
}
throw error;
}
}
// Middleware
function authenticate(req: Request, res: Response, next: NextFunction) {
const authHeader = req.headers.authorization;
if (!authHeader?.startsWith('Bearer ')) {
return res.status(401).json({ error: 'No token provided' });
}
const token = authHeader.substring(7);
try {
const payload = verifyToken(token);
req.user = payload; // Attach user to request
next();
} catch (error) {
return res.status(401).json({ error: 'Invalid token' });
}
}
// Usage
app.get('/api/profile', authenticate, (req, res) => {
res.json({ user: req.user });
});
```
### Pattern 2: Refresh Token Flow
```typescript
interface StoredRefreshToken {
token: string;
userId: string;
expiresAt: Date;
createdAt: Date;
}
class RefreshTokenService {
// Store refresh token in database
async storeRefreshToken(userId: string, refreshToken: string) {
const expiresAt = new Date(Date.now() + 7 * 24 * 60 * 60 * 1000);
await db.refreshTokens.create({
token: await hash(refreshToken), // Hash before storing
userId,
expiresAt,
});
}
// Refresh access token
async refreshAccessToken(refreshToken: string) {
// Verify refresh token
let payload;
try {
payload = jwt.verify(
refreshToken,
process.env.JWT_REFRESH_SECRET!
) as { userId: string };
} catch {
throw new Error('Invalid refresh token');
}
// Check if token exists in database
const storedToken = await db.refreshTokens.findOne({
where: {
token: await hash(refreshToken),
userId: payload.userId,
expiresAt: { $gt: new Date() },
},
});
if (!storedToken) {
throw new Error('Refresh token not found or expired');
}
// Get user
const user = await db.users.findById(payload.userId);
if (!user) {
throw new Error('User not found');
}
// Generate new access token
const accessToken = jwt.sign(
{ userId: user.id, email: user.email, role: user.role },
process.env.JWT_SECRET!,
{ expiresIn: '15m' }
);
return { accessToken };
}
// Revoke refresh token (logout)
async revokeRefreshToken(refreshToken: string) {
await db.refreshTokens.deleteOne({
token: await hash(refreshToken),
});
}
// Revoke all user tokens (logout all devices)
async revokeAllUserTokens(userId: string) {
await db.refreshTokens.deleteMany({ userId });
}
}
// API endpoints
app.post('/api/auth/refresh', async (req, res) => {
const { refreshToken } = req.body;
try {
const { accessToken } = await refreshTokenService
.refreshAccessToken(refreshToken);
res.json({ accessToken });
} catch (error) {
res.status(401).json({ error: 'Invalid refresh token' });
}
});
app.post('/api/auth/logout', authenticate, async (req, res) => {
const { refreshToken } = req.body;
await refreshTokenService.revokeRefreshToken(refreshToken);
res.json({ message: 'Logged out successfully' });
});
```
## Session-Based Authentication
### Pattern 1: Express Session
```typescript
import session from 'express-session';
import RedisStore from 'connect-redis';
import { createClient } from 'redis';
// Setup Redis for session storage
const redisClient = createClient({
url: process.env.REDIS_URL,
});
await redisClient.connect();
app.use(
session({
store: new RedisStore({ client: redisClient }),
secret: process.env.SESSION_SECRET!,
resave: false,
saveUninitialized: false,
cookie: {
secure: process.env.NODE_ENV === 'production', // HTTPS only
httpOnly: true, // No JavaScript access
maxAge: 24 * 60 * 60 * 1000, // 24 hours
sameSite: 'strict', // CSRF protection
},
})
);
// Login
app.post('/api/auth/login', async (req, res) => {
const { email, password } = req.body;
const user = await db.users.findOne({ email });
if (!user || !(await verifyPassword(password, user.passwordHash))) {
return res.status(401).json({ error: 'Invalid credentials' });
}
// Store user in session
req.session.userId = user.id;
req.session.role = user.role;
res.json({ user: { id: user.id, email: user.email, role: user.role } });
});
// Session middleware
function requireAuth(req: Request, res: Response, next: NextFunction) {
if (!req.session.userId) {
return res.status(401).json({ error: 'Not authenticated' });
}
next();
}
// Protected route
app.get('/api/profile', requireAuth, async (req, res) => {
const user = await db.users.findById(req.session.userId);
res.json({ user });
});
// Logout
app.post('/api/auth/logout', (req, res) => {
req.session.destroy((err) => {
if (err) {
return res.status(500).json({ error: 'Logout failed' });
}
res.clearCookie('connect.sid');
res.json({ message: 'Logged out successfully' });
});
});
```
## OAuth2 / Social Login
### Pattern 1: OAuth2 with Passport.js
```typescript
import passport from 'passport';
import { Strategy as GoogleStrategy } from 'passport-google-oauth20';
import { Strategy as GitHubStrategy } from 'passport-github2';
// Google OAuth
passport.use(
new GoogleStrategy(
{
clientID: process.env.GOOGLE_CLIENT_ID!,
clientSecret: process.env.GOOGLE_CLIENT_SECRET!,
callbackURL: '/api/auth/google/callback',
},
async (accessToken, refreshToken, profile, done) => {
try {
// Find or create user
let user = await db.users.findOne({
googleId: profile.id,
});
if (!user) {
user = await db.users.create({
googleId: profile.id,
email: profile.emails?.[0]?.value,
name: profile.displayName,
avatar: profile.photos?.[0]?.value,
});
}
return done(null, user);
} catch (error) {
return done(error, undefined);
}
}
)
);
// Routes
app.get('/api/auth/google', passport.authenticate('google', {
scope: ['profile', 'email'],
}));
app.get(
'/api/auth/google/callback',
passport.authenticate('google', { session: false }),
(req, res) => {
// Generate JWT
const tokens = generateTokens(req.user.id, req.user.email, req.user.role);
// Redirect to frontend with token
res.redirect(`${process.env.FRONTEND_URL}/auth/callback?token=${tokens.accessToken}`);
}
);
```
## Authorization Patterns
### Pattern 1: Role-Based Access Control (RBAC)
```typescript
enum Role {
USER = 'user',
MODERATOR = 'moderator',
ADMIN = 'admin',
}
const roleHierarchy: Record<Role, Role[]> = {
[Role.ADMIN]: [Role.ADMIN, Role.MODERATOR, Role.USER],
[Role.MODERATOR]: [Role.MODERATOR, Role.USER],
[Role.USER]: [Role.USER],
};
function hasRole(userRole: Role, requiredRole: Role): boolean {
return roleHierarchy[userRole].includes(requiredRole);
}
// Middleware
function requireRole(...roles: Role[]) {
return (req: Request, res: Response, next: NextFunction) => {
if (!req.user) {
return res.status(401).json({ error: 'Not authenticated' });
}
if (!roles.some(role => hasRole(req.user.role, role))) {
return res.status(403).json({ error: 'Insufficient permissions' });
}
next();
};
}
// Usage
app.delete('/api/users/:id',
authenticate,
requireRole(Role.ADMIN),
async (req, res) => {
// Only admins can delete users
await db.users.delete(req.params.id);
res.json({ message: 'User deleted' });
}
);
```
### Pattern 2: Permission-Based Access Control
```typescript
enum Permission {
READ_USERS = 'read:users',
WRITE_USERS = 'write:users',
DELETE_USERS = 'delete:users',
READ_POSTS = 'read:posts',
WRITE_POSTS = 'write:posts',
}
const rolePermissions: Record<Role, Permission[]> = {
[Role.USER]: [Permission.READ_POSTS, Permission.WRITE_POSTS],
[Role.MODERATOR]: [
Permission.READ_POSTS,
Permission.WRITE_POSTS,
Permission.READ_USERS,
],
[Role.ADMIN]: Object.values(Permission),
};
function hasPermission(userRole: Role, permission: Permission): boolean {
return rolePermissions[userRole]?.includes(permission) ?? false;
}
function requirePermission(...permissions: Permission[]) {
return (req: Request, res: Response, next: NextFunction) => {
if (!req.user) {
return res.status(401).json({ error: 'Not authenticated' });
}
const hasAllPermissions = permissions.every(permission =>
hasPermission(req.user.role, permission)
);
if (!hasAllPermissions) {
return res.status(403).json({ error: 'Insufficient permissions' });
}
next();
};
}
// Usage
app.get('/api/users',
authenticate,
requirePermission(Permission.READ_USERS),
async (req, res) => {
const users = await db.users.findAll();
res.json({ users });
}
);
```
### Pattern 3: Resource Ownership
```typescript
// Check if user owns resource
async function requireOwnership(
resourceType: 'post' | 'comment',
resourceIdParam: string = 'id'
) {
return async (req: Request, res: Response, next: NextFunction) => {
if (!req.user) {
return res.status(401).json({ error: 'Not authenticated' });
}
const resourceId = req.params[resourceIdParam];
// Admins can access anything
if (req.user.role === Role.ADMIN) {
return next();
}
// Check ownership
let resource;
if (resourceType === 'post') {
resource = await db.posts.findById(resourceId);
} else if (resourceType === 'comment') {
resource = await db.comments.findById(resourceId);
}
if (!resource) {
return res.status(404).json({ error: 'Resource not found' });
}
if (resource.userId !== req.user.userId) {
return res.status(403).json({ error: 'Not authorized' });
}
next();
};
}
// Usage
app.put('/api/posts/:id',
authenticate,
requireOwnership('post'),
async (req, res) => {
// User can only update their own posts
const post = await db.posts.update(req.params.id, req.body);
res.json({ post });
}
);
```
## Security Best Practices
### Pattern 1: Password Security
```typescript
import bcrypt from 'bcrypt';
import { z } from 'zod';
// Password validation schema
const passwordSchema = z.string()
.min(12, 'Password must be at least 12 characters')
.regex(/[A-Z]/, 'Password must contain uppercase letter')
.regex(/[a-z]/, 'Password must contain lowercase letter')
.regex(/[0-9]/, 'Password must contain number')
.regex(/[^A-Za-z0-9]/, 'Password must contain special character');
// Hash password
async function hashPassword(password: string): Promise<string> {
const saltRounds = 12; // 2^12 iterations
return bcrypt.hash(password, saltRounds);
}
// Verify password
async function verifyPassword(
password: string,
hash: string
): Promise<boolean> {
return bcrypt.compare(password, hash);
}
// Registration with password validation
app.post('/api/auth/register', async (req, res) => {
try {
const { email, password } = req.body;
// Validate password
passwordSchema.parse(password);
// Check if user exists
const existingUser = await db.users.findOne({ email });
if (existingUser) {
return res.status(400).json({ error: 'Email already registered' });
}
// Hash password
const passwordHash = await hashPassword(password);
// Create user
const user = await db.users.create({
email,
passwordHash,
});
// Generate tokens
const tokens = generateTokens(user.id, user.email, user.role);
res.status(201).json({
user: { id: user.id, email: user.email },
...tokens,
});
} catch (error) {
if (error instanceof z.ZodError) {
return res.status(400).json({ error: error.errors[0].message });
}
res.status(500).json({ error: 'Registration failed' });
}
});
```
### Pattern 2: Rate Limiting
```typescript
import rateLimit from 'express-rate-limit';
import RedisStore from 'rate-limit-redis';
// Login rate limiter
const loginLimiter = rateLimit({
store: new RedisStore({ client: redisClient }),
windowMs: 15 * 60 * 1000, // 15 minutes
max: 5, // 5 attempts
message: 'Too many login attempts, please try again later',
standardHeaders: true,
legacyHeaders: false,
});
// API rate limiter
const apiLimiter = rateLimit({
windowMs: 60 * 1000, // 1 minute
max: 100, // 100 requests per minute
standardHeaders: true,
});
// Apply to routes
app.post('/api/auth/login', loginLimiter, async (req, res) => {
// Login logic
});
app.use('/api/', apiLimiter);
```
## Best Practices
1. **Never Store Plain Passwords**: Always hash with bcrypt/argon2
2. **Use HTTPS**: Encrypt data in transit
3. **Short-Lived Access Tokens**: 15-30 minutes max
4. **Secure Cookies**: httpOnly, secure, sameSite flags
5. **Validate All Input**: Email format, password strength
6. **Rate Limit Auth Endpoints**: Prevent brute force attacks
7. **Implement CSRF Protection**: For session-based auth
8. **Rotate Secrets Regularly**: JWT secrets, session secrets
9. **Log Security Events**: Login attempts, failed auth
10. **Use MFA When Possible**: Extra security layer
## Common Pitfalls
- **Weak Passwords**: Enforce strong password policies
- **JWT in localStorage**: Vulnerable to XSS, use httpOnly cookies
- **No Token Expiration**: Tokens should expire
- **Client-Side Auth Checks Only**: Always validate server-side
- **Insecure Password Reset**: Use secure tokens with expiration
- **No Rate Limiting**: Vulnerable to brute force
- **Trusting Client Data**: Always validate on server
## Resources
- **references/jwt-best-practices.md**: JWT implementation guide
- **references/oauth2-flows.md**: OAuth2 flow diagrams and examples
- **references/session-security.md**: Secure session management
- **assets/auth-security-checklist.md**: Security review checklist
- **assets/password-policy-template.md**: Password requirements template
- **scripts/token-validator.ts**: JWT validation utility

520
skills/code-review-excellence/SKILL.md Normal file
View File

@@ -0,0 +1,520 @@
---
name: code-review-excellence
description: Master effective code review practices to provide constructive feedback, catch bugs early, and foster knowledge sharing while maintaining team morale. Use when reviewing pull requests, establishing review standards, or mentoring developers.
---
# Code Review Excellence
Transform code reviews from gatekeeping to knowledge sharing through constructive feedback, systematic analysis, and collaborative improvement.
## When to Use This Skill
- Reviewing pull requests and code changes
- Establishing code review standards for teams
- Mentoring junior developers through reviews
- Conducting architecture reviews
- Creating review checklists and guidelines
- Improving team collaboration
- Reducing code review cycle time
- Maintaining code quality standards
## Core Principles
### 1. The Review Mindset
**Goals of Code Review:**
- Catch bugs and edge cases
- Ensure code maintainability
- Share knowledge across team
- Enforce coding standards
- Improve design and architecture
- Build team culture
**Not the Goals:**
- Show off knowledge
- Nitpick formatting (use linters)
- Block progress unnecessarily
- Rewrite to your preference
### 2. Effective Feedback
**Good Feedback is:**
- Specific and actionable
- Educational, not judgmental
- Focused on the code, not the person
- Balanced (praise good work too)
- Prioritized (critical vs nice-to-have)
```markdown
❌ Bad: "This is wrong."
✅ Good: "This could cause a race condition when multiple users
access simultaneously. Consider using a mutex here."
❌ Bad: "Why didn't you use X pattern?"
✅ Good: "Have you considered the Repository pattern? It would
make this easier to test. Here's an example: [link]"
❌ Bad: "Rename this variable."
✅ Good: "[nit] Consider `userCount` instead of `uc` for
clarity. Not blocking if you prefer to keep it."
```
### 3. Review Scope
**What to Review:**
- Logic correctness and edge cases
- Security vulnerabilities
- Performance implications
- Test coverage and quality
- Error handling
- Documentation and comments
- API design and naming
- Architectural fit
**What Not to Review Manually:**
- Code formatting (use Prettier, Black, etc.)
- Import organization
- Linting violations
- Simple typos
## Review Process
### Phase 1: Context Gathering (2-3 minutes)
```markdown
Before diving into code, understand:
1. Read PR description and linked issue
2. Check PR size (>400 lines? Ask to split)
3. Review CI/CD status (tests passing?)
4. Understand the business requirement
5. Note any relevant architectural decisions
```
### Phase 2: High-Level Review (5-10 minutes)
```markdown
1. **Architecture & Design**
- Does the solution fit the problem?
- Are there simpler approaches?
- Is it consistent with existing patterns?
- Will it scale?
2. **File Organization**
- Are new files in the right places?
- Is code grouped logically?
- Are there duplicate files?
3. **Testing Strategy**
- Are there tests?
- Do tests cover edge cases?
- Are tests readable?
```
### Phase 3: Line-by-Line Review (10-20 minutes)
```markdown
For each file:
1. **Logic & Correctness**
- Edge cases handled?
- Off-by-one errors?
- Null/undefined checks?
- Race conditions?
2. **Security**
- Input validation?
- SQL injection risks?
- XSS vulnerabilities?
- Sensitive data exposure?
3. **Performance**
- N+1 queries?
- Unnecessary loops?
- Memory leaks?
- Blocking operations?
4. **Maintainability**
- Clear variable names?
- Functions doing one thing?
- Complex code commented?
- Magic numbers extracted?
```
### Phase 4: Summary & Decision (2-3 minutes)
```markdown
1. Summarize key concerns
2. Highlight what you liked
3. Make clear decision:
- ✅ Approve
- 💬 Comment (minor suggestions)
- 🔄 Request Changes (must address)
4. Offer to pair if complex
```
## Review Techniques
### Technique 1: The Checklist Method
```markdown
## Security Checklist
- [ ] User input validated and sanitized
- [ ] SQL queries use parameterization
- [ ] Authentication/authorization checked
- [ ] Secrets not hardcoded
- [ ] Error messages don't leak info
## Performance Checklist
- [ ] No N+1 queries
- [ ] Database queries indexed
- [ ] Large lists paginated
- [ ] Expensive operations cached
- [ ] No blocking I/O in hot paths
## Testing Checklist
- [ ] Happy path tested
- [ ] Edge cases covered
- [ ] Error cases tested
- [ ] Test names are descriptive
- [ ] Tests are deterministic
```
### Technique 2: The Question Approach
Instead of stating problems, ask questions to encourage thinking:
```markdown
❌ "This will fail if the list is empty."
✅ "What happens if `items` is an empty array?"
❌ "You need error handling here."
✅ "How should this behave if the API call fails?"
❌ "This is inefficient."
✅ "I see this loops through all users. Have we considered
the performance impact with 100k users?"
```
### Technique 3: Suggest, Don't Command
```markdown
## Use Collaborative Language
❌ "You must change this to use async/await"
✅ "Suggestion: async/await might make this more readable:
```typescript
async function fetchUser(id: string) {
const user = await db.query('SELECT * FROM users WHERE id = ?', id);
return user;
}
```
What do you think?"
❌ "Extract this into a function"
✅ "This logic appears in 3 places. Would it make sense to
extract it into a shared utility function?"
```
### Technique 4: Differentiate Severity
```markdown
Use labels to indicate priority:
🔴 [blocking] - Must fix before merge
🟡 [important] - Should fix, discuss if disagree
🟢 [nit] - Nice to have, not blocking
💡 [suggestion] - Alternative approach to consider
📚 [learning] - Educational comment, no action needed
🎉 [praise] - Good work, keep it up!
Example:
"🔴 [blocking] This SQL query is vulnerable to injection.
Please use parameterized queries."
"🟢 [nit] Consider renaming `data` to `userData` for clarity."
"🎉 [praise] Excellent test coverage! This will catch edge cases."
```
## Language-Specific Patterns
### Python Code Review
```python
# Check for Python-specific issues
# ❌ Mutable default arguments
def add_item(item, items=[]): # Bug! Shared across calls
items.append(item)
return items
# ✅ Use None as default
def add_item(item, items=None):
if items is None:
items = []
items.append(item)
return items
# ❌ Catching too broad
try:
result = risky_operation()
except: # Catches everything, even KeyboardInterrupt!
pass
# ✅ Catch specific exceptions
try:
result = risky_operation()
except ValueError as e:
logger.error(f"Invalid value: {e}")
raise
# ❌ Using mutable class attributes
class User:
permissions = [] # Shared across all instances!
# ✅ Initialize in __init__
class User:
def __init__(self):
self.permissions = []
```
### TypeScript/JavaScript Code Review
```typescript
// Check for TypeScript-specific issues
// ❌ Using any defeats type safety
function processData(data: any) { // Avoid any
return data.value;
}
// ✅ Use proper types
interface DataPayload {
value: string;
}
function processData(data: DataPayload) {
return data.value;
}
// ❌ Not handling async errors
async function fetchUser(id: string) {
const response = await fetch(`/api/users/${id}`);
return response.json(); // What if network fails?
}
// ✅ Handle errors properly
async function fetchUser(id: string): Promise<User> {
try {
const response = await fetch(`/api/users/${id}`);
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
return await response.json();
} catch (error) {
console.error('Failed to fetch user:', error);
throw error;
}
}
// ❌ Mutation of props
function UserProfile({ user }: Props) {
user.lastViewed = new Date(); // Mutating prop!
return <div>{user.name}</div>;
}
// ✅ Don't mutate props
function UserProfile({ user, onView }: Props) {
useEffect(() => {
onView(user.id); // Notify parent to update
}, [user.id]);
return <div>{user.name}</div>;
}
```
## Advanced Review Patterns
### Pattern 1: Architectural Review
```markdown
When reviewing significant changes:
1. **Design Document First**
- For large features, request design doc before code
- Review design with team before implementation
- Agree on approach to avoid rework
2. **Review in Stages**
- First PR: Core abstractions and interfaces
- Second PR: Implementation
- Third PR: Integration and tests
- Easier to review, faster to iterate
3. **Consider Alternatives**
- "Have we considered using [pattern/library]?"
- "What's the tradeoff vs. the simpler approach?"
- "How will this evolve as requirements change?"
```
### Pattern 2: Test Quality Review
```typescript
// ❌ Poor test: Implementation detail testing
test('increments counter variable', () => {
const component = render(<Counter />);
const button = component.getByRole('button');
fireEvent.click(button);
expect(component.state.counter).toBe(1); // Testing internal state
});
// ✅ Good test: Behavior testing
test('displays incremented count when clicked', () => {
render(<Counter />);
const button = screen.getByRole('button', { name: /increment/i });
fireEvent.click(button);
expect(screen.getByText('Count: 1')).toBeInTheDocument();
});
// Review questions for tests:
// - Do tests describe behavior, not implementation?
// - Are test names clear and descriptive?
// - Do tests cover edge cases?
// - Are tests independent (no shared state)?
// - Can tests run in any order?
```
### Pattern 3: Security Review
```markdown
## Security Review Checklist
### Authentication & Authorization
- [ ] Is authentication required where needed?
- [ ] Are authorization checks before every action?
- [ ] Is JWT validation proper (signature, expiry)?
- [ ] Are API keys/secrets properly secured?
### Input Validation
- [ ] All user inputs validated?
- [ ] File uploads restricted (size, type)?
- [ ] SQL queries parameterized?
- [ ] XSS protection (escape output)?
### Data Protection
- [ ] Passwords hashed (bcrypt/argon2)?
- [ ] Sensitive data encrypted at rest?
- [ ] HTTPS enforced for sensitive data?
- [ ] PII handled according to regulations?
### Common Vulnerabilities
- [ ] No eval() or similar dynamic execution?
- [ ] No hardcoded secrets?
- [ ] CSRF protection for state-changing operations?
- [ ] Rate limiting on public endpoints?
```
## Giving Difficult Feedback
### Pattern: The Sandwich Method (Modified)
```markdown
Traditional: Praise + Criticism + Praise (feels fake)
Better: Context + Specific Issue + Helpful Solution
Example:
"I noticed the payment processing logic is inline in the
controller. This makes it harder to test and reuse.
[Specific Issue]
The calculateTotal() function mixes tax calculation,
discount logic, and database queries, making it difficult
to unit test and reason about.
[Helpful Solution]
Could we extract this into a PaymentService class? That
would make it testable and reusable. I can pair with you
on this if helpful."
```
### Handling Disagreements
```markdown
When author disagrees with your feedback:
1. **Seek to Understand**
"Help me understand your approach. What led you to
choose this pattern?"
2. **Acknowledge Valid Points**
"That's a good point about X. I hadn't considered that."
3. **Provide Data**
"I'm concerned about performance. Can we add a benchmark
to validate the approach?"
4. **Escalate if Needed**
"Let's get [architect/senior dev] to weigh in on this."
5. **Know When to Let Go**
If it's working and not a critical issue, approve it.
Perfection is the enemy of progress.
```
## Best Practices
1. **Review Promptly**: Within 24 hours, ideally same day
2. **Limit PR Size**: 200-400 lines max for effective review
3. **Review in Time Blocks**: 60 minutes max, take breaks
4. **Use Review Tools**: GitHub, GitLab, or dedicated tools
5. **Automate What You Can**: Linters, formatters, security scans
6. **Build Rapport**: Emoji, praise, and empathy matter
7. **Be Available**: Offer to pair on complex issues
8. **Learn from Others**: Review others' review comments
## Common Pitfalls
- **Perfectionism**: Blocking PRs for minor style preferences
- **Scope Creep**: "While you're at it, can you also..."
- **Inconsistency**: Different standards for different people
- **Delayed Reviews**: Letting PRs sit for days
- **Ghosting**: Requesting changes then disappearing
- **Rubber Stamping**: Approving without actually reviewing
- **Bike Shedding**: Debating trivial details extensively
## Templates
### PR Review Comment Template
```markdown
## Summary
[Brief overview of what was reviewed]
## Strengths
- [What was done well]
- [Good patterns or approaches]
## Required Changes
🔴 [Blocking issue 1]
🔴 [Blocking issue 2]
## Suggestions
💡 [Improvement 1]
💡 [Improvement 2]
## Questions
❓ [Clarification needed on X]
❓ [Alternative approach consideration]
## Verdict
✅ Approve after addressing required changes
```
## Resources
- **references/code-review-best-practices.md**: Comprehensive review guidelines
- **references/common-bugs-checklist.md**: Language-specific bugs to watch for
- **references/security-review-guide.md**: Security-focused review checklist
- **assets/pr-review-template.md**: Standard review comment template
- **assets/review-checklist.md**: Quick reference checklist
- **scripts/pr-analyzer.py**: Analyze PR complexity and suggest reviewers

527
skills/debugging-strategies/SKILL.md Normal file
View File

@@ -0,0 +1,527 @@
---
name: debugging-strategies
description: Master systematic debugging techniques, profiling tools, and root cause analysis to efficiently track down bugs across any codebase or technology stack. Use when investigating bugs, performance issues, or unexpected behavior.
---
# Debugging Strategies
Transform debugging from frustrating guesswork into systematic problem-solving with proven strategies, powerful tools, and methodical approaches.
## When to Use This Skill
- Tracking down elusive bugs
- Investigating performance issues
- Understanding unfamiliar codebases
- Debugging production issues
- Analyzing crash dumps and stack traces
- Profiling application performance
- Investigating memory leaks
- Debugging distributed systems
## Core Principles
### 1. The Scientific Method
**1. Observe**: What's the actual behavior?
**2. Hypothesize**: What could be causing it?
**3. Experiment**: Test your hypothesis
**4. Analyze**: Did it prove/disprove your theory?
**5. Repeat**: Until you find the root cause
### 2. Debugging Mindset
**Don't Assume:**
- "It can't be X" - Yes it can
- "I didn't change Y" - Check anyway
- "It works on my machine" - Find out why
**Do:**
- Reproduce consistently
- Isolate the problem
- Keep detailed notes
- Question everything
- Take breaks when stuck
### 3. Rubber Duck Debugging
Explain your code and problem out loud (to a rubber duck, colleague, or yourself). Often reveals the issue.
## Systematic Debugging Process
### Phase 1: Reproduce
```markdown
## Reproduction Checklist
1. **Can you reproduce it?**
- Always? Sometimes? Randomly?
- Specific conditions needed?
- Can others reproduce it?
2. **Create minimal reproduction**
- Simplify to smallest example
- Remove unrelated code
- Isolate the problem
3. **Document steps**
- Write down exact steps
- Note environment details
- Capture error messages
```
### Phase 2: Gather Information
```markdown
## Information Collection
1. **Error Messages**
- Full stack trace
- Error codes
- Console/log output
2. **Environment**
- OS version
- Language/runtime version
- Dependencies versions
- Environment variables
3. **Recent Changes**
- Git history
- Deployment timeline
- Configuration changes
4. **Scope**
- Affects all users or specific ones?
- All browsers or specific ones?
- Production only or also dev?
```
### Phase 3: Form Hypothesis
```markdown
## Hypothesis Formation
Based on gathered info, ask:
1. **What changed?**
- Recent code changes
- Dependency updates
- Infrastructure changes
2. **What's different?**
- Working vs broken environment
- Working vs broken user
- Before vs after
3. **Where could this fail?**
- Input validation
- Business logic
- Data layer
- External services
```
### Phase 4: Test & Verify
```markdown
## Testing Strategies
1. **Binary Search**
- Comment out half the code
- Narrow down problematic section
- Repeat until found
2. **Add Logging**
- Strategic console.log/print
- Track variable values
- Trace execution flow
3. **Isolate Components**
- Test each piece separately
- Mock dependencies
- Remove complexity
4. **Compare Working vs Broken**
- Diff configurations
- Diff environments
- Diff data
```
## Debugging Tools
### JavaScript/TypeScript Debugging
```typescript
// Chrome DevTools Debugger
function processOrder(order: Order) {
debugger; // Execution pauses here
const total = calculateTotal(order);
console.log('Total:', total);
// Conditional breakpoint
if (order.items.length > 10) {
debugger; // Only breaks if condition true
}
return total;
}
// Console debugging techniques
console.log('Value:', value); // Basic
console.table(arrayOfObjects); // Table format
console.time('operation'); /* code */ console.timeEnd('operation'); // Timing
console.trace(); // Stack trace
console.assert(value > 0, 'Value must be positive'); // Assertion
// Performance profiling
performance.mark('start-operation');
// ... operation code
performance.mark('end-operation');
performance.measure('operation', 'start-operation', 'end-operation');
console.log(performance.getEntriesByType('measure'));
```
**VS Code Debugger Configuration:**
```json
// .vscode/launch.json
{
"version": "0.2.0",
"configurations": [
{
"type": "node",
"request": "launch",
"name": "Debug Program",
"program": "${workspaceFolder}/src/index.ts",
"preLaunchTask": "tsc: build - tsconfig.json",
"outFiles": ["${workspaceFolder}/dist/**/*.js"],
"skipFiles": ["<node_internals>/**"]
},
{
"type": "node",
"request": "launch",
"name": "Debug Tests",
"program": "${workspaceFolder}/node_modules/jest/bin/jest",
"args": ["--runInBand", "--no-cache"],
"console": "integratedTerminal"
}
]
}
```
### Python Debugging
```python
# Built-in debugger (pdb)
import pdb
def calculate_total(items):
total = 0
pdb.set_trace() # Debugger starts here
for item in items:
total += item.price * item.quantity
return total
# Breakpoint (Python 3.7+)
def process_order(order):
breakpoint() # More convenient than pdb.set_trace()
# ... code
# Post-mortem debugging
try:
risky_operation()
except Exception:
import pdb
pdb.post_mortem() # Debug at exception point
# IPython debugging (ipdb)
from ipdb import set_trace
set_trace() # Better interface than pdb
# Logging for debugging
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
def fetch_user(user_id):
logger.debug(f'Fetching user: {user_id}')
user = db.query(User).get(user_id)
logger.debug(f'Found user: {user}')
return user
# Profile performance
import cProfile
import pstats
cProfile.run('slow_function()', 'profile_stats')
stats = pstats.Stats('profile_stats')
stats.sort_stats('cumulative')
stats.print_stats(10) # Top 10 slowest
```
### Go Debugging
```go
// Delve debugger
// Install: go install github.com/go-delve/delve/cmd/dlv@latest
// Run: dlv debug main.go
import (
"fmt"
"runtime"
"runtime/debug"
)
// Print stack trace
func debugStack() {
debug.PrintStack()
}
// Panic recovery with debugging
func processRequest() {
defer func() {
if r := recover(); r != nil {
fmt.Println("Panic:", r)
debug.PrintStack()
}
}()
// ... code that might panic
}
// Memory profiling
import _ "net/http/pprof"
// Visit http://localhost:6060/debug/pprof/
// CPU profiling
import (
"os"
"runtime/pprof"
)
f, _ := os.Create("cpu.prof")
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
// ... code to profile
```
## Advanced Debugging Techniques
### Technique 1: Binary Search Debugging
```bash
# Git bisect for finding regression
git bisect start
git bisect bad # Current commit is bad
git bisect good v1.0.0 # v1.0.0 was good
# Git checks out middle commit
# Test it, then:
git bisect good # if it works
git bisect bad # if it's broken
# Continue until bug found
git bisect reset # when done
```
### Technique 2: Differential Debugging
Compare working vs broken:
```markdown
## What's Different?
| Aspect | Working | Broken |
|--------------|-----------------|-----------------|
| Environment | Development | Production |
| Node version | 18.16.0 | 18.15.0 |
| Data | Empty DB | 1M records |
| User | Admin | Regular user |
| Browser | Chrome | Safari |
| Time | During day | After midnight |
Hypothesis: Time-based issue? Check timezone handling.
```
### Technique 3: Trace Debugging
```typescript
// Function call tracing
function trace(target: any, propertyKey: string, descriptor: PropertyDescriptor) {
const originalMethod = descriptor.value;
descriptor.value = function(...args: any[]) {
console.log(`Calling ${propertyKey} with args:`, args);
const result = originalMethod.apply(this, args);
console.log(`${propertyKey} returned:`, result);
return result;
};
return descriptor;
}
class OrderService {
@trace
calculateTotal(items: Item[]): number {
return items.reduce((sum, item) => sum + item.price, 0);
}
}
```
### Technique 4: Memory Leak Detection
```typescript
// Chrome DevTools Memory Profiler
// 1. Take heap snapshot
// 2. Perform action
// 3. Take another snapshot
// 4. Compare snapshots
// Node.js memory debugging
if (process.memoryUsage().heapUsed > 500 * 1024 * 1024) {
console.warn('High memory usage:', process.memoryUsage());
// Generate heap dump
require('v8').writeHeapSnapshot();
}
// Find memory leaks in tests
let beforeMemory: number;
beforeEach(() => {
beforeMemory = process.memoryUsage().heapUsed;
});
afterEach(() => {
const afterMemory = process.memoryUsage().heapUsed;
const diff = afterMemory - beforeMemory;
if (diff > 10 * 1024 * 1024) { // 10MB threshold
console.warn(`Possible memory leak: ${diff / 1024 / 1024}MB`);
}
});
```
## Debugging Patterns by Issue Type
### Pattern 1: Intermittent Bugs
```markdown
## Strategies for Flaky Bugs
1. **Add extensive logging**
- Log timing information
- Log all state transitions
- Log external interactions
2. **Look for race conditions**
- Concurrent access to shared state
- Async operations completing out of order
- Missing synchronization
3. **Check timing dependencies**
- setTimeout/setInterval
- Promise resolution order
- Animation frame timing
4. **Stress test**
- Run many times
- Vary timing
- Simulate load
```
### Pattern 2: Performance Issues
```markdown
## Performance Debugging
1. **Profile first**
- Don't optimize blindly
- Measure before and after
- Find bottlenecks
2. **Common culprits**
- N+1 queries
- Unnecessary re-renders
- Large data processing
- Synchronous I/O
3. **Tools**
- Browser DevTools Performance tab
- Lighthouse
- Python: cProfile, line_profiler
- Node: clinic.js, 0x
```
### Pattern 3: Production Bugs
```markdown
## Production Debugging
1. **Gather evidence**
- Error tracking (Sentry, Bugsnag)
- Application logs
- User reports
- Metrics/monitoring
2. **Reproduce locally**
- Use production data (anonymized)
- Match environment
- Follow exact steps
3. **Safe investigation**
- Don't change production
- Use feature flags
- Add monitoring/logging
- Test fixes in staging
```
## Best Practices
1. **Reproduce First**: Can't fix what you can't reproduce
2. **Isolate the Problem**: Remove complexity until minimal case
3. **Read Error Messages**: They're usually helpful
4. **Check Recent Changes**: Most bugs are recent
5. **Use Version Control**: Git bisect, blame, history
6. **Take Breaks**: Fresh eyes see better
7. **Document Findings**: Help future you
8. **Fix Root Cause**: Not just symptoms
## Common Debugging Mistakes
- **Making Multiple Changes**: Change one thing at a time
- **Not Reading Error Messages**: Read the full stack trace
- **Assuming It's Complex**: Often it's simple
- **Debug Logging in Prod**: Remove before shipping
- **Not Using Debugger**: console.log isn't always best
- **Giving Up Too Soon**: Persistence pays off
- **Not Testing the Fix**: Verify it actually works
## Quick Debugging Checklist
```markdown
## When Stuck, Check:
- [ ] Spelling errors (typos in variable names)
- [ ] Case sensitivity (fileName vs filename)
- [ ] Null/undefined values
- [ ] Array index off-by-one
- [ ] Async timing (race conditions)
- [ ] Scope issues (closure, hoisting)
- [ ] Type mismatches
- [ ] Missing dependencies
- [ ] Environment variables
- [ ] File paths (absolute vs relative)
- [ ] Cache issues (clear cache)
- [ ] Stale data (refresh database)
```
## Resources
- **references/debugging-tools-guide.md**: Comprehensive tool documentation
- **references/performance-profiling.md**: Performance debugging guide
- **references/production-debugging.md**: Debugging live systems
- **assets/debugging-checklist.md**: Quick reference checklist
- **assets/common-bugs.md**: Common bug patterns
- **scripts/debug-helper.ts**: Debugging utility functions

547
skills/e2e-testing-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,547 @@
---
name: e2e-testing-patterns
description: Master end-to-end testing with Playwright and Cypress to build reliable test suites that catch bugs, improve confidence, and enable fast deployment. Use when implementing E2E tests, debugging flaky tests, or establishing testing standards.
---
# E2E Testing Patterns
Build reliable, fast, and maintainable end-to-end test suites that provide confidence to ship code quickly and catch regressions before users do.
## When to Use This Skill
- Implementing end-to-end test automation
- Debugging flaky or unreliable tests
- Testing critical user workflows
- Setting up CI/CD test pipelines
- Testing across multiple browsers
- Validating accessibility requirements
- Testing responsive designs
- Establishing E2E testing standards
## Core Concepts
### 1. E2E Testing Fundamentals
**What to Test with E2E:**
- Critical user journeys (login, checkout, signup)
- Complex interactions (drag-and-drop, multi-step forms)
- Cross-browser compatibility
- Real API integration
- Authentication flows
**What NOT to Test with E2E:**
- Unit-level logic (use unit tests)
- API contracts (use integration tests)
- Edge cases (too slow)
- Internal implementation details
### 2. Test Philosophy
**The Testing Pyramid:**
```
/\
/E2E\ ← Few, focused on critical paths
/─────\
/Integr\ ← More, test component interactions
/────────\
/Unit Tests\ ← Many, fast, isolated
/────────────\
```
**Best Practices:**
- Test user behavior, not implementation
- Keep tests independent
- Make tests deterministic
- Optimize for speed
- Use data-testid, not CSS selectors
## Playwright Patterns
### Setup and Configuration
```typescript
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
timeout: 30000,
expect: {
timeout: 5000,
},
fullyParallel: true,
forbidOnly: !!process.env.CI,
retries: process.env.CI ? 2 : 0,
workers: process.env.CI ? 1 : undefined,
reporter: [
['html'],
['junit', { outputFile: 'results.xml' }],
],
use: {
baseURL: 'http://localhost:3000',
trace: 'on-first-retry',
screenshot: 'only-on-failure',
video: 'retain-on-failure',
},
projects: [
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
{ name: 'firefox', use: { ...devices['Desktop Firefox'] } },
{ name: 'webkit', use: { ...devices['Desktop Safari'] } },
{ name: 'mobile', use: { ...devices['iPhone 13'] } },
],
});
```
### Pattern 1: Page Object Model
```typescript
// pages/LoginPage.ts
import { Page, Locator } from '@playwright/test';
export class LoginPage {
readonly page: Page;
readonly emailInput: Locator;
readonly passwordInput: Locator;
readonly loginButton: Locator;
readonly errorMessage: Locator;
constructor(page: Page) {
this.page = page;
this.emailInput = page.getByLabel('Email');
this.passwordInput = page.getByLabel('Password');
this.loginButton = page.getByRole('button', { name: 'Login' });
this.errorMessage = page.getByRole('alert');
}
async goto() {
await this.page.goto('/login');
}
async login(email: string, password: string) {
await this.emailInput.fill(email);
await this.passwordInput.fill(password);
await this.loginButton.click();
}
async getErrorMessage(): Promise<string> {
return await this.errorMessage.textContent() ?? '';
}
}
// Test using Page Object
import { test, expect } from '@playwright/test';
import { LoginPage } from './pages/LoginPage';
test('successful login', async ({ page }) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await loginPage.login('user@example.com', 'password123');
await expect(page).toHaveURL('/dashboard');
await expect(page.getByRole('heading', { name: 'Dashboard' }))
.toBeVisible();
});
test('failed login shows error', async ({ page }) => {
const loginPage = new LoginPage(page);
await loginPage.goto();
await loginPage.login('invalid@example.com', 'wrong');
const error = await loginPage.getErrorMessage();
expect(error).toContain('Invalid credentials');
});
```
### Pattern 2: Fixtures for Test Data
```typescript
// fixtures/test-data.ts
import { test as base } from '@playwright/test';
type TestData = {
testUser: {
email: string;
password: string;
name: string;
};
adminUser: {
email: string;
password: string;
};
};
export const test = base.extend<TestData>({
testUser: async ({}, use) => {
const user = {
email: `test-${Date.now()}@example.com`,
password: 'Test123!@#',
name: 'Test User',
};
// Setup: Create user in database
await createTestUser(user);
await use(user);
// Teardown: Clean up user
await deleteTestUser(user.email);
},
adminUser: async ({}, use) => {
await use({
email: 'admin@example.com',
password: process.env.ADMIN_PASSWORD!,
});
},
});
// Usage in tests
import { test } from './fixtures/test-data';
test('user can update profile', async ({ page, testUser }) => {
await page.goto('/login');
await page.getByLabel('Email').fill(testUser.email);
await page.getByLabel('Password').fill(testUser.password);
await page.getByRole('button', { name: 'Login' }).click();
await page.goto('/profile');
await page.getByLabel('Name').fill('Updated Name');
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByText('Profile updated')).toBeVisible();
});
```
### Pattern 3: Waiting Strategies
```typescript
// ❌ Bad: Fixed timeouts
await page.waitForTimeout(3000); // Flaky!
// ✅ Good: Wait for specific conditions
await page.waitForLoadState('networkidle');
await page.waitForURL('/dashboard');
await page.waitForSelector('[data-testid="user-profile"]');
// ✅ Better: Auto-waiting with assertions
await expect(page.getByText('Welcome')).toBeVisible();
await expect(page.getByRole('button', { name: 'Submit' }))
.toBeEnabled();
// Wait for API response
const responsePromise = page.waitForResponse(
response => response.url().includes('/api/users') && response.status() === 200
);
await page.getByRole('button', { name: 'Load Users' }).click();
const response = await responsePromise;
const data = await response.json();
expect(data.users).toHaveLength(10);
// Wait for multiple conditions
await Promise.all([
page.waitForURL('/success'),
page.waitForLoadState('networkidle'),
expect(page.getByText('Payment successful')).toBeVisible(),
]);
```
### Pattern 4: Network Mocking and Interception
```typescript
// Mock API responses
test('displays error when API fails', async ({ page }) => {
await page.route('**/api/users', route => {
route.fulfill({
status: 500,
contentType: 'application/json',
body: JSON.stringify({ error: 'Internal Server Error' }),
});
});
await page.goto('/users');
await expect(page.getByText('Failed to load users')).toBeVisible();
});
// Intercept and modify requests
test('can modify API request', async ({ page }) => {
await page.route('**/api/users', async route => {
const request = route.request();
const postData = JSON.parse(request.postData() || '{}');
// Modify request
postData.role = 'admin';
await route.continue({
postData: JSON.stringify(postData),
});
});
// Test continues...
});
// Mock third-party services
test('payment flow with mocked Stripe', async ({ page }) => {
await page.route('**/api/stripe/**', route => {
route.fulfill({
status: 200,
body: JSON.stringify({
id: 'mock_payment_id',
status: 'succeeded',
}),
});
});
// Test payment flow with mocked response
});
```
## Cypress Patterns
### Setup and Configuration
```typescript
// cypress.config.ts
import { defineConfig } from 'cypress';
export default defineConfig({
e2e: {
baseUrl: 'http://localhost:3000',
viewportWidth: 1280,
viewportHeight: 720,
video: false,
screenshotOnRunFailure: true,
defaultCommandTimeout: 10000,
requestTimeout: 10000,
setupNodeEvents(on, config) {
// Implement node event listeners
},
},
});
```
### Pattern 1: Custom Commands
```typescript
// cypress/support/commands.ts
declare global {
namespace Cypress {
interface Chainable {
login(email: string, password: string): Chainable<void>;
createUser(userData: UserData): Chainable<User>;
dataCy(value: string): Chainable<JQuery<HTMLElement>>;
}
}
}
Cypress.Commands.add('login', (email: string, password: string) => {
cy.visit('/login');
cy.get('[data-testid="email"]').type(email);
cy.get('[data-testid="password"]').type(password);
cy.get('[data-testid="login-button"]').click();
cy.url().should('include', '/dashboard');
});
Cypress.Commands.add('createUser', (userData: UserData) => {
return cy.request('POST', '/api/users', userData)
.its('body');
});
Cypress.Commands.add('dataCy', (value: string) => {
return cy.get(`[data-cy="${value}"]`);
});
// Usage
cy.login('user@example.com', 'password');
cy.dataCy('submit-button').click();
```
### Pattern 2: Cypress Intercept
```typescript
// Mock API calls
cy.intercept('GET', '/api/users', {
statusCode: 200,
body: [
{ id: 1, name: 'John' },
{ id: 2, name: 'Jane' },
],
}).as('getUsers');
cy.visit('/users');
cy.wait('@getUsers');
cy.get('[data-testid="user-list"]').children().should('have.length', 2);
// Modify responses
cy.intercept('GET', '/api/users', (req) => {
req.reply((res) => {
// Modify response
res.body.users = res.body.users.slice(0, 5);
res.send();
});
});
// Simulate slow network
cy.intercept('GET', '/api/data', (req) => {
req.reply((res) => {
res.delay(3000); // 3 second delay
res.send();
});
});
```
## Advanced Patterns
### Pattern 1: Visual Regression Testing
```typescript
// With Playwright
import { test, expect } from '@playwright/test';
test('homepage looks correct', async ({ page }) => {
await page.goto('/');
await expect(page).toHaveScreenshot('homepage.png', {
fullPage: true,
maxDiffPixels: 100,
});
});
test('button in all states', async ({ page }) => {
await page.goto('/components');
const button = page.getByRole('button', { name: 'Submit' });
// Default state
await expect(button).toHaveScreenshot('button-default.png');
// Hover state
await button.hover();
await expect(button).toHaveScreenshot('button-hover.png');
// Disabled state
await button.evaluate(el => el.setAttribute('disabled', 'true'));
await expect(button).toHaveScreenshot('button-disabled.png');
});
```
### Pattern 2: Parallel Testing with Sharding
```typescript
// playwright.config.ts
export default defineConfig({
projects: [
{
name: 'shard-1',
use: { ...devices['Desktop Chrome'] },
grepInvert: /@slow/,
shard: { current: 1, total: 4 },
},
{
name: 'shard-2',
use: { ...devices['Desktop Chrome'] },
shard: { current: 2, total: 4 },
},
// ... more shards
],
});
// Run in CI
// npx playwright test --shard=1/4
// npx playwright test --shard=2/4
```
### Pattern 3: Accessibility Testing
```typescript
// Install: npm install @axe-core/playwright
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test('page should not have accessibility violations', async ({ page }) => {
await page.goto('/');
const accessibilityScanResults = await new AxeBuilder({ page })
.exclude('#third-party-widget')
.analyze();
expect(accessibilityScanResults.violations).toEqual([]);
});
test('form is accessible', async ({ page }) => {
await page.goto('/signup');
const results = await new AxeBuilder({ page })
.include('form')
.analyze();
expect(results.violations).toEqual([]);
});
```
## Best Practices
1. **Use Data Attributes**: `data-testid` or `data-cy` for stable selectors
2. **Avoid Brittle Selectors**: Don't rely on CSS classes or DOM structure
3. **Test User Behavior**: Click, type, see - not implementation details
4. **Keep Tests Independent**: Each test should run in isolation
5. **Clean Up Test Data**: Create and destroy test data in each test
6. **Use Page Objects**: Encapsulate page logic
7. **Meaningful Assertions**: Check actual user-visible behavior
8. **Optimize for Speed**: Mock when possible, parallel execution
```typescript
// ❌ Bad selectors
cy.get('.btn.btn-primary.submit-button').click();
cy.get('div > form > div:nth-child(2) > input').type('text');
// ✅ Good selectors
cy.getByRole('button', { name: 'Submit' }).click();
cy.getByLabel('Email address').type('user@example.com');
cy.get('[data-testid="email-input"]').type('user@example.com');
```
## Common Pitfalls
- **Flaky Tests**: Use proper waits, not fixed timeouts
- **Slow Tests**: Mock external APIs, use parallel execution
- **Over-Testing**: Don't test every edge case with E2E
- **Coupled Tests**: Tests should not depend on each other
- **Poor Selectors**: Avoid CSS classes and nth-child
- **No Cleanup**: Clean up test data after each test
- **Testing Implementation**: Test user behavior, not internals
## Debugging Failing Tests
```typescript
// Playwright debugging
// 1. Run in headed mode
npx playwright test --headed
// 2. Run in debug mode
npx playwright test --debug
// 3. Use trace viewer
await page.screenshot({ path: 'screenshot.png' });
await page.video()?.saveAs('video.webm');
// 4. Add test.step for better reporting
test('checkout flow', async ({ page }) => {
await test.step('Add item to cart', async () => {
await page.goto('/products');
await page.getByRole('button', { name: 'Add to Cart' }).click();
});
await test.step('Proceed to checkout', async () => {
await page.goto('/cart');
await page.getByRole('button', { name: 'Checkout' }).click();
});
});
// 5. Inspect page state
await page.pause(); // Pauses execution, opens inspector
```
## Resources
- **references/playwright-best-practices.md**: Playwright-specific patterns
- **references/cypress-best-practices.md**: Cypress-specific patterns
- **references/flaky-test-debugging.md**: Debugging unreliable tests
- **assets/e2e-testing-checklist.md**: What to test with E2E
- **assets/selector-strategies.md**: Finding reliable selectors
- **scripts/test-analyzer.ts**: Analyze test flakiness and duration

636
skills/error-handling-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,636 @@
---
name: error-handling-patterns
description: Master error handling patterns across languages including exceptions, Result types, error propagation, and graceful degradation to build resilient applications. Use when implementing error handling, designing APIs, or improving application reliability.
---
# Error Handling Patterns
Build resilient applications with robust error handling strategies that gracefully handle failures and provide excellent debugging experiences.
## When to Use This Skill
- Implementing error handling in new features
- Designing error-resilient APIs
- Debugging production issues
- Improving application reliability
- Creating better error messages for users and developers
- Implementing retry and circuit breaker patterns
- Handling async/concurrent errors
- Building fault-tolerant distributed systems
## Core Concepts
### 1. Error Handling Philosophies
**Exceptions vs Result Types:**
- **Exceptions**: Traditional try-catch, disrupts control flow
- **Result Types**: Explicit success/failure, functional approach
- **Error Codes**: C-style, requires discipline
- **Option/Maybe Types**: For nullable values
**When to Use Each:**
- Exceptions: Unexpected errors, exceptional conditions
- Result Types: Expected errors, validation failures
- Panics/Crashes: Unrecoverable errors, programming bugs
### 2. Error Categories
**Recoverable Errors:**
- Network timeouts
- Missing files
- Invalid user input
- API rate limits
**Unrecoverable Errors:**
- Out of memory
- Stack overflow
- Programming bugs (null pointer, etc.)
## Language-Specific Patterns
### Python Error Handling
**Custom Exception Hierarchy:**
```python
class ApplicationError(Exception):
"""Base exception for all application errors."""
def __init__(self, message: str, code: str = None, details: dict = None):
super().__init__(message)
self.code = code
self.details = details or {}
self.timestamp = datetime.utcnow()
class ValidationError(ApplicationError):
"""Raised when validation fails."""
pass
class NotFoundError(ApplicationError):
"""Raised when resource not found."""
pass
class ExternalServiceError(ApplicationError):
"""Raised when external service fails."""
def __init__(self, message: str, service: str, **kwargs):
super().__init__(message, **kwargs)
self.service = service
# Usage
def get_user(user_id: str) -> User:
user = db.query(User).filter_by(id=user_id).first()
if not user:
raise NotFoundError(
f"User not found",
code="USER_NOT_FOUND",
details={"user_id": user_id}
)
return user
```
**Context Managers for Cleanup:**
```python
from contextlib import contextmanager
@contextmanager
def database_transaction(session):
"""Ensure transaction is committed or rolled back."""
try:
yield session
session.commit()
except Exception as e:
session.rollback()
raise
finally:
session.close()
# Usage
with database_transaction(db.session) as session:
user = User(name="Alice")
session.add(user)
# Automatic commit or rollback
```
**Retry with Exponential Backoff:**
```python
import time
from functools import wraps
from typing import TypeVar, Callable
T = TypeVar('T')
def retry(
max_attempts: int = 3,
backoff_factor: float = 2.0,
exceptions: tuple = (Exception,)
):
"""Retry decorator with exponential backoff."""
def decorator(func: Callable[..., T]) -> Callable[..., T]:
@wraps(func)
def wrapper(*args, **kwargs) -> T:
last_exception = None
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except exceptions as e:
last_exception = e
if attempt < max_attempts - 1:
sleep_time = backoff_factor ** attempt
time.sleep(sleep_time)
continue
raise
raise last_exception
return wrapper
return decorator
# Usage
@retry(max_attempts=3, exceptions=(NetworkError,))
def fetch_data(url: str) -> dict:
response = requests.get(url, timeout=5)
response.raise_for_status()
return response.json()
```
### TypeScript/JavaScript Error Handling
**Custom Error Classes:**
```typescript
// Custom error classes
class ApplicationError extends Error {
constructor(
message: string,
public code: string,
public statusCode: number = 500,
public details?: Record<string, any>
) {
super(message);
this.name = this.constructor.name;
Error.captureStackTrace(this, this.constructor);
}
}
class ValidationError extends ApplicationError {
constructor(message: string, details?: Record<string, any>) {
super(message, 'VALIDATION_ERROR', 400, details);
}
}
class NotFoundError extends ApplicationError {
constructor(resource: string, id: string) {
super(
`${resource} not found`,
'NOT_FOUND',
404,
{ resource, id }
);
}
}
// Usage
function getUser(id: string): User {
const user = users.find(u => u.id === id);
if (!user) {
throw new NotFoundError('User', id);
}
return user;
}
```
**Result Type Pattern:**
```typescript
// Result type for explicit error handling
type Result<T, E = Error> =
| { ok: true; value: T }
| { ok: false; error: E };
// Helper functions
function Ok<T>(value: T): Result<T, never> {
return { ok: true, value };
}
function Err<E>(error: E): Result<never, E> {
return { ok: false, error };
}
// Usage
function parseJSON<T>(json: string): Result<T, SyntaxError> {
try {
const value = JSON.parse(json) as T;
return Ok(value);
} catch (error) {
return Err(error as SyntaxError);
}
}
// Consuming Result
const result = parseJSON<User>(userJson);
if (result.ok) {
console.log(result.value.name);
} else {
console.error('Parse failed:', result.error.message);
}
// Chaining Results
function chain<T, U, E>(
result: Result<T, E>,
fn: (value: T) => Result<U, E>
): Result<U, E> {
return result.ok ? fn(result.value) : result;
}
```
**Async Error Handling:**
```typescript
// Async/await with proper error handling
async function fetchUserOrders(userId: string): Promise<Order[]> {
try {
const user = await getUser(userId);
const orders = await getOrders(user.id);
return orders;
} catch (error) {
if (error instanceof NotFoundError) {
return []; // Return empty array for not found
}
if (error instanceof NetworkError) {
// Retry logic
return retryFetchOrders(userId);
}
// Re-throw unexpected errors
throw error;
}
}
// Promise error handling
function fetchData(url: string): Promise<Data> {
return fetch(url)
.then(response => {
if (!response.ok) {
throw new NetworkError(`HTTP ${response.status}`);
}
return response.json();
})
.catch(error => {
console.error('Fetch failed:', error);
throw error;
});
}
```
### Rust Error Handling
**Result and Option Types:**
```rust
use std::fs::File;
use std::io::{self, Read};
// Result type for operations that can fail
fn read_file(path: &str) -> Result<String, io::Error> {
let mut file = File::open(path)?; // ? operator propagates errors
let mut contents = String::new();
file.read_to_string(&mut contents)?;
Ok(contents)
}
// Custom error types
#[derive(Debug)]
enum AppError {
Io(io::Error),
Parse(std::num::ParseIntError),
NotFound(String),
Validation(String),
}
impl From<io::Error> for AppError {
fn from(error: io::Error) -> Self {
AppError::Io(error)
}
}
// Using custom error type
fn read_number_from_file(path: &str) -> Result<i32, AppError> {
let contents = read_file(path)?; // Auto-converts io::Error
let number = contents.trim().parse()
.map_err(AppError::Parse)?; // Explicitly convert ParseIntError
Ok(number)
}
// Option for nullable values
fn find_user(id: &str) -> Option<User> {
users.iter().find(|u| u.id == id).cloned()
}
// Combining Option and Result
fn get_user_age(id: &str) -> Result<u32, AppError> {
find_user(id)
.ok_or_else(|| AppError::NotFound(id.to_string()))
.map(|user| user.age)
}
```
### Go Error Handling
**Explicit Error Returns:**
```go
// Basic error handling
func getUser(id string) (*User, error) {
user, err := db.QueryUser(id)
if err != nil {
return nil, fmt.Errorf("failed to query user: %w", err)
}
if user == nil {
return nil, errors.New("user not found")
}
return user, nil
}
// Custom error types
type ValidationError struct {
Field string
Message string
}
func (e *ValidationError) Error() string {
return fmt.Sprintf("validation failed for %s: %s", e.Field, e.Message)
}
// Sentinel errors for comparison
var (
ErrNotFound = errors.New("not found")
ErrUnauthorized = errors.New("unauthorized")
ErrInvalidInput = errors.New("invalid input")
)
// Error checking
user, err := getUser("123")
if err != nil {
if errors.Is(err, ErrNotFound) {
// Handle not found
} else {
// Handle other errors
}
}
// Error wrapping and unwrapping
func processUser(id string) error {
user, err := getUser(id)
if err != nil {
return fmt.Errorf("process user failed: %w", err)
}
// Process user
return nil
}
// Unwrap errors
err := processUser("123")
if err != nil {
var valErr *ValidationError
if errors.As(err, &valErr) {
fmt.Printf("Validation error: %s\n", valErr.Field)
}
}
```
## Universal Patterns
### Pattern 1: Circuit Breaker
Prevent cascading failures in distributed systems.
```python
from enum import Enum
from datetime import datetime, timedelta
from typing import Callable, TypeVar
T = TypeVar('T')
class CircuitState(Enum):
CLOSED = "closed" # Normal operation
OPEN = "open" # Failing, reject requests
HALF_OPEN = "half_open" # Testing if recovered
class CircuitBreaker:
def __init__(
self,
failure_threshold: int = 5,
timeout: timedelta = timedelta(seconds=60),
success_threshold: int = 2
):
self.failure_threshold = failure_threshold
self.timeout = timeout
self.success_threshold = success_threshold
self.failure_count = 0
self.success_count = 0
self.state = CircuitState.CLOSED
self.last_failure_time = None
def call(self, func: Callable[[], T]) -> T:
if self.state == CircuitState.OPEN:
if datetime.now() - self.last_failure_time > self.timeout:
self.state = CircuitState.HALF_OPEN
self.success_count = 0
else:
raise Exception("Circuit breaker is OPEN")
try:
result = func()
self.on_success()
return result
except Exception as e:
self.on_failure()
raise
def on_success(self):
self.failure_count = 0
if self.state == CircuitState.HALF_OPEN:
self.success_count += 1
if self.success_count >= self.success_threshold:
self.state = CircuitState.CLOSED
self.success_count = 0
def on_failure(self):
self.failure_count += 1
self.last_failure_time = datetime.now()
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
# Usage
circuit_breaker = CircuitBreaker()
def fetch_data():
return circuit_breaker.call(lambda: external_api.get_data())
```
### Pattern 2: Error Aggregation
Collect multiple errors instead of failing on first error.
```typescript
class ErrorCollector {
private errors: Error[] = [];
add(error: Error): void {
this.errors.push(error);
}
hasErrors(): boolean {
return this.errors.length > 0;
}
getErrors(): Error[] {
return [...this.errors];
}
throw(): never {
if (this.errors.length === 1) {
throw this.errors[0];
}
throw new AggregateError(
this.errors,
`${this.errors.length} errors occurred`
);
}
}
// Usage: Validate multiple fields
function validateUser(data: any): User {
const errors = new ErrorCollector();
if (!data.email) {
errors.add(new ValidationError('Email is required'));
} else if (!isValidEmail(data.email)) {
errors.add(new ValidationError('Email is invalid'));
}
if (!data.name || data.name.length < 2) {
errors.add(new ValidationError('Name must be at least 2 characters'));
}
if (!data.age || data.age < 18) {
errors.add(new ValidationError('Age must be 18 or older'));
}
if (errors.hasErrors()) {
errors.throw();
}
return data as User;
}
```
### Pattern 3: Graceful Degradation
Provide fallback functionality when errors occur.
```python
from typing import Optional, Callable, TypeVar
T = TypeVar('T')
def with_fallback(
primary: Callable[[], T],
fallback: Callable[[], T],
log_error: bool = True
) -> T:
"""Try primary function, fall back to fallback on error."""
try:
return primary()
except Exception as e:
if log_error:
logger.error(f"Primary function failed: {e}")
return fallback()
# Usage
def get_user_profile(user_id: str) -> UserProfile:
return with_fallback(
primary=lambda: fetch_from_cache(user_id),
fallback=lambda: fetch_from_database(user_id)
)
# Multiple fallbacks
def get_exchange_rate(currency: str) -> float:
return (
try_function(lambda: api_provider_1.get_rate(currency))
or try_function(lambda: api_provider_2.get_rate(currency))
or try_function(lambda: cache.get_rate(currency))
or DEFAULT_RATE
)
def try_function(func: Callable[[], Optional[T]]) -> Optional[T]:
try:
return func()
except Exception:
return None
```
## Best Practices
1. **Fail Fast**: Validate input early, fail quickly
2. **Preserve Context**: Include stack traces, metadata, timestamps
3. **Meaningful Messages**: Explain what happened and how to fix it
4. **Log Appropriately**: Error = log, expected failure = don't spam logs
5. **Handle at Right Level**: Catch where you can meaningfully handle
6. **Clean Up Resources**: Use try-finally, context managers, defer
7. **Don't Swallow Errors**: Log or re-throw, don't silently ignore
8. **Type-Safe Errors**: Use typed errors when possible
```python
# Good error handling example
def process_order(order_id: str) -> Order:
"""Process order with comprehensive error handling."""
try:
# Validate input
if not order_id:
raise ValidationError("Order ID is required")
# Fetch order
order = db.get_order(order_id)
if not order:
raise NotFoundError("Order", order_id)
# Process payment
try:
payment_result = payment_service.charge(order.total)
except PaymentServiceError as e:
# Log and wrap external service error
logger.error(f"Payment failed for order {order_id}: {e}")
raise ExternalServiceError(
f"Payment processing failed",
service="payment_service",
details={"order_id": order_id, "amount": order.total}
) from e
# Update order
order.status = "completed"
order.payment_id = payment_result.id
db.save(order)
return order
except ApplicationError:
# Re-raise known application errors
raise
except Exception as e:
# Log unexpected errors
logger.exception(f"Unexpected error processing order {order_id}")
raise ApplicationError(
"Order processing failed",
code="INTERNAL_ERROR"
) from e
```
## Common Pitfalls
- **Catching Too Broadly**: `except Exception` hides bugs
- **Empty Catch Blocks**: Silently swallowing errors
- **Logging and Re-throwing**: Creates duplicate log entries
- **Not Cleaning Up**: Forgetting to close files, connections
- **Poor Error Messages**: "Error occurred" is not helpful
- **Returning Error Codes**: Use exceptions or Result types
- **Ignoring Async Errors**: Unhandled promise rejections
## Resources
- **references/exception-hierarchy-design.md**: Designing error class hierarchies
- **references/error-recovery-strategies.md**: Recovery patterns for different scenarios
- **references/async-error-handling.md**: Handling errors in concurrent code
- **assets/error-handling-checklist.md**: Review checklist for error handling
- **assets/error-message-guide.md**: Writing helpful error messages
- **scripts/error-analyzer.py**: Analyze error patterns in logs

400
skills/git-advanced-workflows/SKILL.md Normal file
View File

@@ -0,0 +1,400 @@
---
name: git-advanced-workflows
description: Master advanced Git workflows including rebasing, cherry-picking, bisect, worktrees, and reflog to maintain clean history and recover from any situation. Use when managing complex Git histories, collaborating on feature branches, or troubleshooting repository issues.
---
# Git Advanced Workflows
Master advanced Git techniques to maintain clean history, collaborate effectively, and recover from any situation with confidence.
## When to Use This Skill
- Cleaning up commit history before merging
- Applying specific commits across branches
- Finding commits that introduced bugs
- Working on multiple features simultaneously
- Recovering from Git mistakes or lost commits
- Managing complex branch workflows
- Preparing clean PRs for review
- Synchronizing diverged branches
## Core Concepts
### 1. Interactive Rebase
Interactive rebase is the Swiss Army knife of Git history editing.
**Common Operations:**
- `pick`: Keep commit as-is
- `reword`: Change commit message
- `edit`: Amend commit content
- `squash`: Combine with previous commit
- `fixup`: Like squash but discard message
- `drop`: Remove commit entirely
**Basic Usage:**
```bash
# Rebase last 5 commits
git rebase -i HEAD~5
# Rebase all commits on current branch
git rebase -i $(git merge-base HEAD main)
# Rebase onto specific commit
git rebase -i abc123
```
### 2. Cherry-Picking
Apply specific commits from one branch to another without merging entire branches.
```bash
# Cherry-pick single commit
git cherry-pick abc123
# Cherry-pick range of commits (exclusive start)
git cherry-pick abc123..def456
# Cherry-pick without committing (stage changes only)
git cherry-pick -n abc123
# Cherry-pick and edit commit message
git cherry-pick -e abc123
```
### 3. Git Bisect
Binary search through commit history to find the commit that introduced a bug.
```bash
# Start bisect
git bisect start
# Mark current commit as bad
git bisect bad
# Mark known good commit
git bisect good v1.0.0
# Git will checkout middle commit - test it
# Then mark as good or bad
git bisect good # or: git bisect bad
# Continue until bug found
# When done
git bisect reset
```
**Automated Bisect:**
```bash
# Use script to test automatically
git bisect start HEAD v1.0.0
git bisect run ./test.sh
# test.sh should exit 0 for good, 1-127 (except 125) for bad
```
### 4. Worktrees
Work on multiple branches simultaneously without stashing or switching.
```bash
# List existing worktrees
git worktree list
# Add new worktree for feature branch
git worktree add ../project-feature feature/new-feature
# Add worktree and create new branch
git worktree add -b bugfix/urgent ../project-hotfix main
# Remove worktree
git worktree remove ../project-feature
# Prune stale worktrees
git worktree prune
```
### 5. Reflog
Your safety net - tracks all ref movements, even deleted commits.
```bash
# View reflog
git reflog
# View reflog for specific branch
git reflog show feature/branch
# Restore deleted commit
git reflog
# Find commit hash
git checkout abc123
git branch recovered-branch
# Restore deleted branch
git reflog
git branch deleted-branch abc123
```
## Practical Workflows
### Workflow 1: Clean Up Feature Branch Before PR
```bash
# Start with feature branch
git checkout feature/user-auth
# Interactive rebase to clean history
git rebase -i main
# Example rebase operations:
# - Squash "fix typo" commits
# - Reword commit messages for clarity
# - Reorder commits logically
# - Drop unnecessary commits
# Force push cleaned branch (safe if no one else is using it)
git push --force-with-lease origin feature/user-auth
```
### Workflow 2: Apply Hotfix to Multiple Releases
```bash
# Create fix on main
git checkout main
git commit -m "fix: critical security patch"
# Apply to release branches
git checkout release/2.0
git cherry-pick abc123
git checkout release/1.9
git cherry-pick abc123
# Handle conflicts if they arise
git cherry-pick --continue
# or
git cherry-pick --abort
```
### Workflow 3: Find Bug Introduction
```bash
# Start bisect
git bisect start
git bisect bad HEAD
git bisect good v2.1.0
# Git checks out middle commit - run tests
npm test
# If tests fail
git bisect bad
# If tests pass
git bisect good
# Git will automatically checkout next commit to test
# Repeat until bug found
# Automated version
git bisect start HEAD v2.1.0
git bisect run npm test
```
### Workflow 4: Multi-Branch Development
```bash
# Main project directory
cd ~/projects/myapp
# Create worktree for urgent bugfix
git worktree add ../myapp-hotfix hotfix/critical-bug
# Work on hotfix in separate directory
cd ../myapp-hotfix
# Make changes, commit
git commit -m "fix: resolve critical bug"
git push origin hotfix/critical-bug
# Return to main work without interruption
cd ~/projects/myapp
git fetch origin
git cherry-pick hotfix/critical-bug
# Clean up when done
git worktree remove ../myapp-hotfix
```
### Workflow 5: Recover from Mistakes
```bash
# Accidentally reset to wrong commit
git reset --hard HEAD~5 # Oh no!
# Use reflog to find lost commits
git reflog
# Output shows:
# abc123 HEAD@{0}: reset: moving to HEAD~5
# def456 HEAD@{1}: commit: my important changes
# Recover lost commits
git reset --hard def456
# Or create branch from lost commit
git branch recovery def456
```
## Advanced Techniques
### Rebase vs Merge Strategy
**When to Rebase:**
- Cleaning up local commits before pushing
- Keeping feature branch up-to-date with main
- Creating linear history for easier review
**When to Merge:**
- Integrating completed features into main
- Preserving exact history of collaboration
- Public branches used by others
```bash
# Update feature branch with main changes (rebase)
git checkout feature/my-feature
git fetch origin
git rebase origin/main
# Handle conflicts
git status
# Fix conflicts in files
git add .
git rebase --continue
# Or merge instead
git merge origin/main
```
### Autosquash Workflow
Automatically squash fixup commits during rebase.
```bash
# Make initial commit
git commit -m "feat: add user authentication"
# Later, fix something in that commit
# Stage changes
git commit --fixup HEAD # or specify commit hash
# Make more changes
git commit --fixup abc123
# Rebase with autosquash
git rebase -i --autosquash main
# Git automatically marks fixup commits
```
### Split Commit
Break one commit into multiple logical commits.
```bash
# Start interactive rebase
git rebase -i HEAD~3
# Mark commit to split with 'edit'
# Git will stop at that commit
# Reset commit but keep changes
git reset HEAD^
# Stage and commit in logical chunks
git add file1.py
git commit -m "feat: add validation"
git add file2.py
git commit -m "feat: add error handling"
# Continue rebase
git rebase --continue
```
### Partial Cherry-Pick
Cherry-pick only specific files from a commit.
```bash
# Show files in commit
git show --name-only abc123
# Checkout specific files from commit
git checkout abc123 -- path/to/file1.py path/to/file2.py
# Stage and commit
git commit -m "cherry-pick: apply specific changes from abc123"
```
## Best Practices
1. **Always Use --force-with-lease**: Safer than --force, prevents overwriting others' work
2. **Rebase Only Local Commits**: Don't rebase commits that have been pushed and shared
3. **Descriptive Commit Messages**: Future you will thank present you
4. **Atomic Commits**: Each commit should be a single logical change
5. **Test Before Force Push**: Ensure history rewrite didn't break anything
6. **Keep Reflog Aware**: Remember reflog is your safety net for 90 days
7. **Branch Before Risky Operations**: Create backup branch before complex rebases
```bash
# Safe force push
git push --force-with-lease origin feature/branch
# Create backup before risky operation
git branch backup-branch
git rebase -i main
# If something goes wrong
git reset --hard backup-branch
```
## Common Pitfalls
- **Rebasing Public Branches**: Causes history conflicts for collaborators
- **Force Pushing Without Lease**: Can overwrite teammate's work
- **Losing Work in Rebase**: Resolve conflicts carefully, test after rebase
- **Forgetting Worktree Cleanup**: Orphaned worktrees consume disk space
- **Not Backing Up Before Experiment**: Always create safety branch
- **Bisect on Dirty Working Directory**: Commit or stash before bisecting
## Recovery Commands
```bash
# Abort operations in progress
git rebase --abort
git merge --abort
git cherry-pick --abort
git bisect reset
# Restore file to version from specific commit
git restore --source=abc123 path/to/file
# Undo last commit but keep changes
git reset --soft HEAD^
# Undo last commit and discard changes
git reset --hard HEAD^
# Recover deleted branch (within 90 days)
git reflog
git branch recovered-branch abc123
```
## Resources
- **references/git-rebase-guide.md**: Deep dive into interactive rebase
- **references/git-conflict-resolution.md**: Advanced conflict resolution strategies
- **references/git-history-rewriting.md**: Safely rewriting Git history
- **assets/git-workflow-checklist.md**: Pre-PR cleanup checklist
- **assets/git-aliases.md**: Useful Git aliases for advanced workflows
- **scripts/git-clean-branches.sh**: Clean up merged and stale branches

622
skills/monorepo-management/SKILL.md Normal file
View File

@@ -0,0 +1,622 @@
---
name: monorepo-management
description: Master monorepo management with Turborepo, Nx, and pnpm workspaces to build efficient, scalable multi-package repositories with optimized builds and dependency management. Use when setting up monorepos, optimizing builds, or managing shared dependencies.
---
# Monorepo Management
Build efficient, scalable monorepos that enable code sharing, consistent tooling, and atomic changes across multiple packages and applications.
## When to Use This Skill
- Setting up new monorepo projects
- Migrating from multi-repo to monorepo
- Optimizing build and test performance
- Managing shared dependencies
- Implementing code sharing strategies
- Setting up CI/CD for monorepos
- Versioning and publishing packages
- Debugging monorepo-specific issues
## Core Concepts
### 1. Why Monorepos?
**Advantages:**
- Shared code and dependencies
- Atomic commits across projects
- Consistent tooling and standards
- Easier refactoring
- Simplified dependency management
- Better code visibility
**Challenges:**
- Build performance at scale
- CI/CD complexity
- Access control
- Large Git repository
### 2. Monorepo Tools
**Package Managers:**
- pnpm workspaces (recommended)
- npm workspaces
- Yarn workspaces
**Build Systems:**
- Turborepo (recommended for most)
- Nx (feature-rich, complex)
- Lerna (older, maintenance mode)
## Turborepo Setup
### Initial Setup
```bash
# Create new monorepo
npx create-turbo@latest my-monorepo
cd my-monorepo
# Structure:
# apps/
# web/ - Next.js app
# docs/ - Documentation site
# packages/
# ui/ - Shared UI components
# config/ - Shared configurations
# tsconfig/ - Shared TypeScript configs
# turbo.json - Turborepo configuration
# package.json - Root package.json
```
### Configuration
```json
// turbo.json
{
"$schema": "https://turbo.build/schema.json",
"globalDependencies": ["**/.env.*local"],
"pipeline": {
"build": {
"dependsOn": ["^build"],
"outputs": ["dist/**", ".next/**", "!.next/cache/**"]
},
"test": {
"dependsOn": ["build"],
"outputs": ["coverage/**"]
},
"lint": {
"outputs": []
},
"dev": {
"cache": false,
"persistent": true
},
"type-check": {
"dependsOn": ["^build"],
"outputs": []
}
}
}
```
```json
// package.json (root)
{
"name": "my-monorepo",
"private": true,
"workspaces": [
"apps/*",
"packages/*"
],
"scripts": {
"build": "turbo run build",
"dev": "turbo run dev",
"test": "turbo run test",
"lint": "turbo run lint",
"format": "prettier --write \"**/*.{ts,tsx,md}\"",
"clean": "turbo run clean && rm -rf node_modules"
},
"devDependencies": {
"turbo": "^1.10.0",
"prettier": "^3.0.0",
"typescript": "^5.0.0"
},
"packageManager": "pnpm@8.0.0"
}
```
### Package Structure
```json
// packages/ui/package.json
{
"name": "@repo/ui",
"version": "0.0.0",
"private": true,
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"import": "./dist/index.js",
"types": "./dist/index.d.ts"
},
"./button": {
"import": "./dist/button.js",
"types": "./dist/button.d.ts"
}
},
"scripts": {
"build": "tsup src/index.ts --format esm,cjs --dts",
"dev": "tsup src/index.ts --format esm,cjs --dts --watch",
"lint": "eslint src/",
"type-check": "tsc --noEmit"
},
"devDependencies": {
"@repo/tsconfig": "workspace:*",
"tsup": "^7.0.0",
"typescript": "^5.0.0"
},
"dependencies": {
"react": "^18.2.0"
}
}
```
## pnpm Workspaces
### Setup
```yaml
# pnpm-workspace.yaml
packages:
- 'apps/*'
- 'packages/*'
- 'tools/*'
```
```json
// .npmrc
# Hoist shared dependencies
shamefully-hoist=true
# Strict peer dependencies
auto-install-peers=true
strict-peer-dependencies=true
# Performance
store-dir=~/.pnpm-store
```
### Dependency Management
```bash
# Install dependency in specific package
pnpm add react --filter @repo/ui
pnpm add -D typescript --filter @repo/ui
# Install workspace dependency
pnpm add @repo/ui --filter web
# Install in all packages
pnpm add -D eslint -w
# Update all dependencies
pnpm update -r
# Remove dependency
pnpm remove react --filter @repo/ui
```
### Scripts
```bash
# Run script in specific package
pnpm --filter web dev
pnpm --filter @repo/ui build
# Run in all packages
pnpm -r build
pnpm -r test
# Run in parallel
pnpm -r --parallel dev
# Filter by pattern
pnpm --filter "@repo/*" build
pnpm --filter "...web" build # Build web and dependencies
```
## Nx Monorepo
### Setup
```bash
# Create Nx monorepo
npx create-nx-workspace@latest my-org
# Generate applications
nx generate @nx/react:app my-app
nx generate @nx/next:app my-next-app
# Generate libraries
nx generate @nx/react:lib ui-components
nx generate @nx/js:lib utils
```
### Configuration
```json
// nx.json
{
"extends": "nx/presets/npm.json",
"$schema": "./node_modules/nx/schemas/nx-schema.json",
"targetDefaults": {
"build": {
"dependsOn": ["^build"],
"inputs": ["production", "^production"],
"cache": true
},
"test": {
"inputs": ["default", "^production", "{workspaceRoot}/jest.preset.js"],
"cache": true
},
"lint": {
"inputs": ["default", "{workspaceRoot}/.eslintrc.json"],
"cache": true
}
},
"namedInputs": {
"default": ["{projectRoot}/**/*", "sharedGlobals"],
"production": [
"default",
"!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)",
"!{projectRoot}/tsconfig.spec.json"
],
"sharedGlobals": []
}
}
```
### Running Tasks
```bash
# Run task for specific project
nx build my-app
nx test ui-components
nx lint utils
# Run for affected projects
nx affected:build
nx affected:test --base=main
# Visualize dependencies
nx graph
# Run in parallel
nx run-many --target=build --all --parallel=3
```
## Shared Configurations
### TypeScript Configuration
```json
// packages/tsconfig/base.json
{
"compilerOptions": {
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"module": "ESNext",
"moduleResolution": "bundler",
"resolveJsonModule": true,
"isolatedModules": true,
"incremental": true,
"declaration": true
},
"exclude": ["node_modules"]
}
// packages/tsconfig/react.json
{
"extends": "./base.json",
"compilerOptions": {
"jsx": "react-jsx",
"lib": ["ES2022", "DOM", "DOM.Iterable"]
}
}
// apps/web/tsconfig.json
{
"extends": "@repo/tsconfig/react.json",
"compilerOptions": {
"outDir": "dist",
"rootDir": "src"
},
"include": ["src"],
"exclude": ["node_modules", "dist"]
}
```
### ESLint Configuration
```javascript
// packages/config/eslint-preset.js
module.exports = {
extends: [
'eslint:recommended',
'plugin:@typescript-eslint/recommended',
'plugin:react/recommended',
'plugin:react-hooks/recommended',
'prettier',
],
plugins: ['@typescript-eslint', 'react', 'react-hooks'],
parser: '@typescript-eslint/parser',
parserOptions: {
ecmaVersion: 2022,
sourceType: 'module',
ecmaFeatures: {
jsx: true,
},
},
settings: {
react: {
version: 'detect',
},
},
rules: {
'@typescript-eslint/no-unused-vars': 'error',
'react/react-in-jsx-scope': 'off',
},
};
// apps/web/.eslintrc.js
module.exports = {
extends: ['@repo/config/eslint-preset'],
rules: {
// App-specific rules
},
};
```
## Code Sharing Patterns
### Pattern 1: Shared UI Components
```typescript
// packages/ui/src/button.tsx
import * as React from 'react';
export interface ButtonProps {
variant?: 'primary' | 'secondary';
children: React.ReactNode;
onClick?: () => void;
}
export function Button({ variant = 'primary', children, onClick }: ButtonProps) {
return (
<button
className={`btn btn-${variant}`}
onClick={onClick}
>
{children}
</button>
);
}
// packages/ui/src/index.ts
export { Button, type ButtonProps } from './button';
export { Input, type InputProps } from './input';
// apps/web/src/app.tsx
import { Button } from '@repo/ui';
export function App() {
return <Button variant="primary">Click me</Button>;
}
```
### Pattern 2: Shared Utilities
```typescript
// packages/utils/src/string.ts
export function capitalize(str: string): string {
return str.charAt(0).toUpperCase() + str.slice(1);
}
export function truncate(str: string, length: number): string {
return str.length > length ? str.slice(0, length) + '...' : str;
}
// packages/utils/src/index.ts
export * from './string';
export * from './array';
export * from './date';
// Usage in apps
import { capitalize, truncate } from '@repo/utils';
```
### Pattern 3: Shared Types
```typescript
// packages/types/src/user.ts
export interface User {
id: string;
email: string;
name: string;
role: 'admin' | 'user';
}
export interface CreateUserInput {
email: string;
name: string;
password: string;
}
// Used in both frontend and backend
import type { User, CreateUserInput } from '@repo/types';
```
## Build Optimization
### Turborepo Caching
```json
// turbo.json
{
"pipeline": {
"build": {
// Build depends on dependencies being built first
"dependsOn": ["^build"],
// Cache these outputs
"outputs": ["dist/**", ".next/**"],
// Cache based on these inputs (default: all files)
"inputs": ["src/**/*.tsx", "src/**/*.ts", "package.json"]
},
"test": {
// Run tests in parallel, don't depend on build
"cache": true,
"outputs": ["coverage/**"]
}
}
}
```
### Remote Caching
```bash
# Turborepo Remote Cache (Vercel)
npx turbo login
npx turbo link
# Custom remote cache
# turbo.json
{
"remoteCache": {
"signature": true,
"enabled": true
}
}
```
## CI/CD for Monorepos
### GitHub Actions
```yaml
# .github/workflows/ci.yml
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0 # For Nx affected commands
- uses: pnpm/action-setup@v2
with:
version: 8
- uses: actions/setup-node@v3
with:
node-version: 18
cache: 'pnpm'
- name: Install dependencies
run: pnpm install --frozen-lockfile
- name: Build
run: pnpm turbo run build
- name: Test
run: pnpm turbo run test
- name: Lint
run: pnpm turbo run lint
- name: Type check
run: pnpm turbo run type-check
```
### Deploy Affected Only
```yaml
# Deploy only changed apps
- name: Deploy affected apps
run: |
if pnpm nx affected:apps --base=origin/main --head=HEAD | grep -q "web"; then
echo "Deploying web app"
pnpm --filter web deploy
fi
```
## Best Practices
1. **Consistent Versioning**: Lock dependency versions across workspace
2. **Shared Configs**: Centralize ESLint, TypeScript, Prettier configs
3. **Dependency Graph**: Keep it acyclic, avoid circular dependencies
4. **Cache Effectively**: Configure inputs/outputs correctly
5. **Type Safety**: Share types between frontend/backend
6. **Testing Strategy**: Unit tests in packages, E2E in apps
7. **Documentation**: README in each package
8. **Release Strategy**: Use changesets for versioning
## Common Pitfalls
- **Circular Dependencies**: A depends on B, B depends on A
- **Phantom Dependencies**: Using deps not in package.json
- **Incorrect Cache Inputs**: Missing files in Turborepo inputs
- **Over-Sharing**: Sharing code that should be separate
- **Under-Sharing**: Duplicating code across packages
- **Large Monorepos**: Without proper tooling, builds slow down
## Publishing Packages
```bash
# Using Changesets
pnpm add -Dw @changesets/cli
pnpm changeset init
# Create changeset
pnpm changeset
# Version packages
pnpm changeset version
# Publish
pnpm changeset publish
```
```yaml
# .github/workflows/release.yml
- name: Create Release Pull Request or Publish
uses: changesets/action@v1
with:
publish: pnpm release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
```
## Resources
- **references/turborepo-guide.md**: Comprehensive Turborepo documentation
- **references/nx-guide.md**: Nx monorepo patterns
- **references/pnpm-workspaces.md**: pnpm workspace features
- **assets/monorepo-checklist.md**: Setup checklist
- **assets/migration-guide.md**: Multi-repo to monorepo migration
- **scripts/dependency-graph.ts**: Visualize package dependencies

493
skills/sql-optimization-patterns/SKILL.md Normal file
View File

@@ -0,0 +1,493 @@
---
name: sql-optimization-patterns
description: Master SQL query optimization, indexing strategies, and EXPLAIN analysis to dramatically improve database performance and eliminate slow queries. Use when debugging slow queries, designing database schemas, or optimizing application performance.
---
# SQL Optimization Patterns
Transform slow database queries into lightning-fast operations through systematic optimization, proper indexing, and query plan analysis.
## When to Use This Skill
- Debugging slow-running queries
- Designing performant database schemas
- Optimizing application response times
- Reducing database load and costs
- Improving scalability for growing datasets
- Analyzing EXPLAIN query plans
- Implementing efficient indexes
- Resolving N+1 query problems
## Core Concepts
### 1. Query Execution Plans (EXPLAIN)
Understanding EXPLAIN output is fundamental to optimization.
**PostgreSQL EXPLAIN:**
```sql
-- Basic explain
EXPLAIN SELECT * FROM users WHERE email = 'user@example.com';
-- With actual execution stats
EXPLAIN ANALYZE
SELECT * FROM users WHERE email = 'user@example.com';
-- Verbose output with more details
EXPLAIN (ANALYZE, BUFFERS, VERBOSE)
SELECT u.*, o.order_total
FROM users u
JOIN orders o ON u.id = o.user_id
WHERE u.created_at > NOW() - INTERVAL '30 days';
```
**Key Metrics to Watch:**
- **Seq Scan**: Full table scan (usually slow for large tables)
- **Index Scan**: Using index (good)
- **Index Only Scan**: Using index without touching table (best)
- **Nested Loop**: Join method (okay for small datasets)
- **Hash Join**: Join method (good for larger datasets)
- **Merge Join**: Join method (good for sorted data)
- **Cost**: Estimated query cost (lower is better)
- **Rows**: Estimated rows returned
- **Actual Time**: Real execution time
### 2. Index Strategies
Indexes are the most powerful optimization tool.
**Index Types:**
- **B-Tree**: Default, good for equality and range queries
- **Hash**: Only for equality (=) comparisons
- **GIN**: Full-text search, array queries, JSONB
- **GiST**: Geometric data, full-text search
- **BRIN**: Block Range INdex for very large tables with correlation
```sql
-- Standard B-Tree index
CREATE INDEX idx_users_email ON users(email);
-- Composite index (order matters!)
CREATE INDEX idx_orders_user_status ON orders(user_id, status);
-- Partial index (index subset of rows)
CREATE INDEX idx_active_users ON users(email)
WHERE status = 'active';
-- Expression index
CREATE INDEX idx_users_lower_email ON users(LOWER(email));
-- Covering index (include additional columns)
CREATE INDEX idx_users_email_covering ON users(email)
INCLUDE (name, created_at);
-- Full-text search index
CREATE INDEX idx_posts_search ON posts
USING GIN(to_tsvector('english', title || ' ' || body));
-- JSONB index
CREATE INDEX idx_metadata ON events USING GIN(metadata);
```
### 3. Query Optimization Patterns
**Avoid SELECT \*:**
```sql
-- Bad: Fetches unnecessary columns
SELECT * FROM users WHERE id = 123;
-- Good: Fetch only what you need
SELECT id, email, name FROM users WHERE id = 123;
```
**Use WHERE Clause Efficiently:**
```sql
-- Bad: Function prevents index usage
SELECT * FROM users WHERE LOWER(email) = 'user@example.com';
-- Good: Create functional index or use exact match
CREATE INDEX idx_users_email_lower ON users(LOWER(email));
-- Then:
SELECT * FROM users WHERE LOWER(email) = 'user@example.com';
-- Or store normalized data
SELECT * FROM users WHERE email = 'user@example.com';
```
**Optimize JOINs:**
```sql
-- Bad: Cartesian product then filter
SELECT u.name, o.total
FROM users u, orders o
WHERE u.id = o.user_id AND u.created_at > '2024-01-01';
-- Good: Filter before join
SELECT u.name, o.total
FROM users u
JOIN orders o ON u.id = o.user_id
WHERE u.created_at > '2024-01-01';
-- Better: Filter both tables
SELECT u.name, o.total
FROM (SELECT * FROM users WHERE created_at > '2024-01-01') u
JOIN orders o ON u.id = o.user_id;
```
## Optimization Patterns
### Pattern 1: Eliminate N+1 Queries
**Problem: N+1 Query Anti-Pattern**
```python
# Bad: Executes N+1 queries
users = db.query("SELECT * FROM users LIMIT 10")
for user in users:
orders = db.query("SELECT * FROM orders WHERE user_id = ?", user.id)
# Process orders
```
**Solution: Use JOINs or Batch Loading**
```sql
-- Solution 1: JOIN
SELECT
u.id, u.name,
o.id as order_id, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.id IN (1, 2, 3, 4, 5);
-- Solution 2: Batch query
SELECT * FROM orders
WHERE user_id IN (1, 2, 3, 4, 5);
```
```python
# Good: Single query with JOIN or batch load
# Using JOIN
results = db.query("""
SELECT u.id, u.name, o.id as order_id, o.total
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE u.id IN (1, 2, 3, 4, 5)
""")
# Or batch load
users = db.query("SELECT * FROM users LIMIT 10")
user_ids = [u.id for u in users]
orders = db.query(
"SELECT * FROM orders WHERE user_id IN (?)",
user_ids
)
# Group orders by user_id
orders_by_user = {}
for order in orders:
orders_by_user.setdefault(order.user_id, []).append(order)
```
### Pattern 2: Optimize Pagination
**Bad: OFFSET on Large Tables**
```sql
-- Slow for large offsets
SELECT * FROM users
ORDER BY created_at DESC
LIMIT 20 OFFSET 100000; -- Very slow!
```
**Good: Cursor-Based Pagination**
```sql
-- Much faster: Use cursor (last seen ID)
SELECT * FROM users
WHERE created_at < '2024-01-15 10:30:00' -- Last cursor
ORDER BY created_at DESC
LIMIT 20;
-- With composite sorting
SELECT * FROM users
WHERE (created_at, id) < ('2024-01-15 10:30:00', 12345)
ORDER BY created_at DESC, id DESC
LIMIT 20;
-- Requires index
CREATE INDEX idx_users_cursor ON users(created_at DESC, id DESC);
```
### Pattern 3: Aggregate Efficiently
**Optimize COUNT Queries:**
```sql
-- Bad: Counts all rows
SELECT COUNT(*) FROM orders; -- Slow on large tables
-- Good: Use estimates for approximate counts
SELECT reltuples::bigint AS estimate
FROM pg_class
WHERE relname = 'orders';
-- Good: Filter before counting
SELECT COUNT(*) FROM orders
WHERE created_at > NOW() - INTERVAL '7 days';
-- Better: Use index-only scan
CREATE INDEX idx_orders_created ON orders(created_at);
SELECT COUNT(*) FROM orders
WHERE created_at > NOW() - INTERVAL '7 days';
```
**Optimize GROUP BY:**
```sql
-- Bad: Group by then filter
SELECT user_id, COUNT(*) as order_count
FROM orders
GROUP BY user_id
HAVING COUNT(*) > 10;
-- Better: Filter first, then group (if possible)
SELECT user_id, COUNT(*) as order_count
FROM orders
WHERE status = 'completed'
GROUP BY user_id
HAVING COUNT(*) > 10;
-- Best: Use covering index
CREATE INDEX idx_orders_user_status ON orders(user_id, status);
```
### Pattern 4: Subquery Optimization
**Transform Correlated Subqueries:**
```sql
-- Bad: Correlated subquery (runs for each row)
SELECT u.name, u.email,
(SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) as order_count
FROM users u;
-- Good: JOIN with aggregation
SELECT u.name, u.email, COUNT(o.id) as order_count
FROM users u
LEFT JOIN orders o ON o.user_id = u.id
GROUP BY u.id, u.name, u.email;
-- Better: Use window functions
SELECT DISTINCT ON (u.id)
u.name, u.email,
COUNT(o.id) OVER (PARTITION BY u.id) as order_count
FROM users u
LEFT JOIN orders o ON o.user_id = u.id;
```
**Use CTEs for Clarity:**
```sql
-- Using Common Table Expressions
WITH recent_users AS (
SELECT id, name, email
FROM users
WHERE created_at > NOW() - INTERVAL '30 days'
),
user_order_counts AS (
SELECT user_id, COUNT(*) as order_count
FROM orders
WHERE created_at > NOW() - INTERVAL '30 days'
GROUP BY user_id
)
SELECT ru.name, ru.email, COALESCE(uoc.order_count, 0) as orders
FROM recent_users ru
LEFT JOIN user_order_counts uoc ON ru.id = uoc.user_id;
```
### Pattern 5: Batch Operations
**Batch INSERT:**
```sql
-- Bad: Multiple individual inserts
INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com');
INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com');
INSERT INTO users (name, email) VALUES ('Carol', 'carol@example.com');
-- Good: Batch insert
INSERT INTO users (name, email) VALUES
('Alice', 'alice@example.com'),
('Bob', 'bob@example.com'),
('Carol', 'carol@example.com');
-- Better: Use COPY for bulk inserts (PostgreSQL)
COPY users (name, email) FROM '/tmp/users.csv' CSV HEADER;
```
**Batch UPDATE:**
```sql
-- Bad: Update in loop
UPDATE users SET status = 'active' WHERE id = 1;
UPDATE users SET status = 'active' WHERE id = 2;
-- ... repeat for many IDs
-- Good: Single UPDATE with IN clause
UPDATE users
SET status = 'active'
WHERE id IN (1, 2, 3, 4, 5, ...);
-- Better: Use temporary table for large batches
CREATE TEMP TABLE temp_user_updates (id INT, new_status VARCHAR);
INSERT INTO temp_user_updates VALUES (1, 'active'), (2, 'active'), ...;
UPDATE users u
SET status = t.new_status
FROM temp_user_updates t
WHERE u.id = t.id;
```
## Advanced Techniques
### Materialized Views
Pre-compute expensive queries.
```sql
-- Create materialized view
CREATE MATERIALIZED VIEW user_order_summary AS
SELECT
u.id,
u.name,
COUNT(o.id) as total_orders,
SUM(o.total) as total_spent,
MAX(o.created_at) as last_order_date
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
GROUP BY u.id, u.name;
-- Add index to materialized view
CREATE INDEX idx_user_summary_spent ON user_order_summary(total_spent DESC);
-- Refresh materialized view
REFRESH MATERIALIZED VIEW user_order_summary;
-- Concurrent refresh (PostgreSQL)
REFRESH MATERIALIZED VIEW CONCURRENTLY user_order_summary;
-- Query materialized view (very fast)
SELECT * FROM user_order_summary
WHERE total_spent > 1000
ORDER BY total_spent DESC;
```
### Partitioning
Split large tables for better performance.
```sql
-- Range partitioning by date (PostgreSQL)
CREATE TABLE orders (
id SERIAL,
user_id INT,
total DECIMAL,
created_at TIMESTAMP
) PARTITION BY RANGE (created_at);
-- Create partitions
CREATE TABLE orders_2024_q1 PARTITION OF orders
FOR VALUES FROM ('2024-01-01') TO ('2024-04-01');
CREATE TABLE orders_2024_q2 PARTITION OF orders
FOR VALUES FROM ('2024-04-01') TO ('2024-07-01');
-- Queries automatically use appropriate partition
SELECT * FROM orders
WHERE created_at BETWEEN '2024-02-01' AND '2024-02-28';
-- Only scans orders_2024_q1 partition
```
### Query Hints and Optimization
```sql
-- Force index usage (MySQL)
SELECT * FROM users
USE INDEX (idx_users_email)
WHERE email = 'user@example.com';
-- Parallel query (PostgreSQL)
SET max_parallel_workers_per_gather = 4;
SELECT * FROM large_table WHERE condition;
-- Join hints (PostgreSQL)
SET enable_nestloop = OFF; -- Force hash or merge join
```
## Best Practices
1. **Index Selectively**: Too many indexes slow down writes
2. **Monitor Query Performance**: Use slow query logs
3. **Keep Statistics Updated**: Run ANALYZE regularly
4. **Use Appropriate Data Types**: Smaller types = better performance
5. **Normalize Thoughtfully**: Balance normalization vs performance
6. **Cache Frequently Accessed Data**: Use application-level caching
7. **Connection Pooling**: Reuse database connections
8. **Regular Maintenance**: VACUUM, ANALYZE, rebuild indexes
```sql
-- Update statistics
ANALYZE users;
ANALYZE VERBOSE orders;
-- Vacuum (PostgreSQL)
VACUUM ANALYZE users;
VACUUM FULL users; -- Reclaim space (locks table)
-- Reindex
REINDEX INDEX idx_users_email;
REINDEX TABLE users;
```
## Common Pitfalls
- **Over-Indexing**: Each index slows down INSERT/UPDATE/DELETE
- **Unused Indexes**: Waste space and slow writes
- **Missing Indexes**: Slow queries, full table scans
- **Implicit Type Conversion**: Prevents index usage
- **OR Conditions**: Can't use indexes efficiently
- **LIKE with Leading Wildcard**: `LIKE '%abc'` can't use index
- **Function in WHERE**: Prevents index usage unless functional index exists
## Monitoring Queries
```sql
-- Find slow queries (PostgreSQL)
SELECT query, calls, total_time, mean_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;
-- Find missing indexes (PostgreSQL)
SELECT
schemaname,
tablename,
seq_scan,
seq_tup_read,
idx_scan,
seq_tup_read / seq_scan AS avg_seq_tup_read
FROM pg_stat_user_tables
WHERE seq_scan > 0
ORDER BY seq_tup_read DESC
LIMIT 10;
-- Find unused indexes (PostgreSQL)
SELECT
schemaname,
tablename,
indexname,
idx_scan,
idx_tup_read,
idx_tup_fetch
FROM pg_stat_user_indexes
WHERE idx_scan = 0
ORDER BY pg_relation_size(indexrelid) DESC;
```
## Resources
- **references/postgres-optimization-guide.md**: PostgreSQL-specific optimization
- **references/mysql-optimization-guide.md**: MySQL/MariaDB optimization
- **references/query-plan-analysis.md**: Deep dive into EXPLAIN plans
- **assets/index-strategy-checklist.md**: When and how to create indexes
- **assets/query-optimization-checklist.md**: Step-by-step optimization guide
- **scripts/analyze-slow-queries.sql**: Identify slow queries in your database
- **scripts/index-recommendations.sql**: Generate index recommendations