Initial commit

2025-11-29 18:47:11 +08:00
commit 8d68fa43fc
13 changed files with 4244 additions and 0 deletions
--- a/commands/db-operations.md
+++ b/commands/db-operations.md
@@ -0,0 +1,523 @@
+# Database Operations Command
+
+Manage database operations for the ExFabrica Agentic Factory project, including migrations, seeding, backup, restore, and schema management using Drizzle ORM and PostgreSQL.
+
+## Usage
+
+```
+/db-operations <operation> [--env=<environment>] [--force]
+```
+
+## Parameters
+
+- **operation** (required): Database operation to perform
+  - `migrate` - Run pending database migrations
+  - `migrate:create` - Create a new migration file
+  - `migrate:rollback` - Rollback the last migration
+  - `seed` - Seed database with test/demo data
+  - `reset` - Drop all tables and recreate schema
+  - `backup` - Create database backup
+  - `restore` - Restore from backup
+  - `status` - Show migration and database status
+  - `validate` - Validate schema against models
+- **--env** (optional): Target environment (dev, test, production). Defaults to current environment.
+- **--force** (optional): Force operation without confirmation (use with caution)
+
+## Operations
+
+### 1. Migrations
+
+#### Run Migrations
+```
+/db-operations migrate
+```
+
+Applies all pending migrations to the database in order.
+
+**Process**:
+1. Check database connection
+2. Retrieve list of applied migrations
+3. Identify pending migrations
+4. Execute migrations in sequence
+5. Record successful migrations
+6. Rollback on error
+
+**Output**:
+```
+🗃️  Running Database Migrations
+================================
+
+Database: exfabrica_af_dev
+Environment: development
+
+Pending migrations:
+  ✓ 001_create_users_table.sql
+  ✓ 002_create_projects_table.sql
+  ✓ 003_add_user_roles.sql
+
+All migrations completed successfully! (3.2s)
+```
+
+#### Create New Migration
+```
+/db-operations migrate:create add_organizations_table
+```
+
+Creates a new migration file with timestamp.
+
+**Generated File**: `apps/backend/drizzle/migrations/20251029123456_add_organizations_table.sql`
+
+#### Rollback Migration
+```
+/db-operations migrate:rollback
+```
+
+Rolls back the most recently applied migration.
+
+**⚠️ WARNING**: Use with extreme caution in production!
+
+### 2. Database Seeding
+
+#### Seed Development Data
+```
+/db-operations seed
+```
+
+Populates the database with test/demo data for development.
+
+**Seed Data Includes**:
+- Test user accounts (admin, regular users)
+- Sample projects and workflows
+- Demo organizations
+- Test API keys
+- Sample data for each entity type
+
+**Process**:
+1. Check if database is empty or can be seeded
+2. Clear existing seed data (if applicable)
+3. Insert seed data in dependency order
+4. Verify data integrity
+5. Report seeded record counts
+
+**Output**:
+```
+🌱 Seeding Database
+===================
+
+Environment: development
+
+Seeding data:
+  ✓ Users: 10 records created
+  ✓ Organizations: 3 records created
+  ✓ Projects: 15 records created
+  ✓ Workflows: 23 records created
+  ✓ API Keys: 5 records created
+
+Database seeded successfully! (2.7s)
+
+Test Accounts:
+- admin@exfabrica.com / Admin123!
+- user@exfabrica.com / User123!
+```
+
+### 3. Database Reset
+
+```
+/db-operations reset
+```
+
+**⚠️ DESTRUCTIVE OPERATION**: Drops all tables and recreates the schema.
+
+**Confirmation Required** (unless --force is used):
+```
+⚠️  WARNING: This will delete ALL data in the database!
+
+Database: exfabrica_af_dev
+Environment: development
+
+Are you sure you want to continue? (yes/no):
+```
+
+**Process**:
+1. Request explicit confirmation
+2. Create backup before reset (unless skipped)
+3. Drop all tables in reverse dependency order
+4. Recreate schema from migrations
+5. Optionally seed with fresh data
+
+**Use Cases**:
+- Reset development database to clean state
+- Clear test database between test runs
+- Fix corrupted schema during development
+
+### 4. Database Backup
+
+```
+/db-operations backup
+```
+
+Creates a full backup of the current database.
+
+**Backup Location**: `backups/db/exfabrica_af_dev_20251029_123456.sql`
+
+**Backup Includes**:
+- Complete schema (tables, indexes, constraints)
+- All data
+- Sequences and serial values
+- Custom types and functions
+
+**Process**:
+1. Generate backup filename with timestamp
+2. Use `pg_dump` to create SQL backup
+3. Compress backup file (gzip)
+4. Verify backup integrity
+5. Report backup location and size
+
+**Output**:
+```
+💾 Creating Database Backup
+===========================
+
+Database: exfabrica_af_dev
+Backup file: backups/db/exfabrica_af_dev_20251029_123456.sql.gz
+
+✓ Backup created successfully
+✓ Size: 12.4 MB (compressed)
+✓ Records: ~45,000
+
+Backup location:
+  c:\Users\nicol\Source\ExFabrica\ExFabrica\EAF\backups\db\exfabrica_af_dev_20251029_123456.sql.gz
+```
+
+### 5. Database Restore
+
+```
+/db-operations restore backups/db/exfabrica_af_dev_20251029_123456.sql.gz
+```
+
+Restores database from a backup file.
+
+**⚠️ WARNING**: This will replace all current data!
+
+**Process**:
+1. Verify backup file exists and is valid
+2. Request confirmation
+3. Create safety backup of current state
+4. Drop existing schema (optional)
+5. Restore from backup file
+6. Verify restoration success
+
+### 6. Database Status
+
+```
+/db-operations status
+```
+
+Shows comprehensive database and migration status.
+
+**Output**:
+```
+📊 Database Status
+==================
+
+Connection: ✓ Connected
+Database: exfabrica_af_dev
+Host: localhost:5432
+Version: PostgreSQL 15.3
+
+Migrations:
+  Applied: 23
+  Pending: 0
+  Last migration: 20251028_add_workflow_permissions.sql
+
+Schema:
+  Tables: 12
+  Indexes: 34
+  Constraints: 28
+
+Data Summary:
+  Users: 145 records
+  Projects: 67 records
+  Workflows: 234 records
+  Organizations: 8 records
+
+Size:
+  Database: 124 MB
+  Largest table: workflows (45 MB)
+
+Health: ✓ All checks passed
+```
+
+### 7. Schema Validation
+
+```
+/db-operations validate
+```
+
+Validates that the database schema matches Drizzle ORM model definitions.
+
+**Checks**:
+- All models have corresponding tables
+- Column types match model definitions
+- Indexes are correctly created
+- Foreign key constraints are in place
+- Default values are set correctly
+
+**Output**:
+```
+✓ All models validated successfully
+⚠ Warning: Index 'idx_users_email' missing on users table
+⚠ Warning: Column 'updated_at' type mismatch (expected: timestamptz, found: timestamp)
+```
+
+## Environment Configuration
+
+### Development Environment
+```bash
+# .env.development
+DATABASE_URL=postgresql://dev:devpass@localhost:5432/exfabrica_af_dev
+```
+
+### Test Environment
+```bash
+# .env.test
+DATABASE_URL=postgresql://test:testpass@localhost:5442/exfabrica_af_test
+```
+
+### Production Environment
+```bash
+# .env.production
+DATABASE_URL=postgresql://prod:prodpass@db.example.com:5432/exfabrica_af_prod
+```
+
+## Examples
+
+### Development Workflow
+
+```bash
+# Start fresh with clean database
+/db-operations reset --force
+/db-operations seed
+
+# Make schema changes in code
+# Generate migration
+/db-operations migrate:create add_new_feature
+
+# Apply migration
+/db-operations migrate
+
+# Verify
+/db-operations status
+```
+
+### Production Deployment
+
+```bash
+# Create backup before migration
+/db-operations backup --env=production
+
+# Apply migrations
+/db-operations migrate --env=production
+
+# Verify
+/db-operations status --env=production
+```
+
+### Testing Setup
+
+```bash
+# Reset test database
+/db-operations reset --env=test --force
+
+# Seed with test data
+/db-operations seed --env=test
+
+# Run tests
+/test-all backend
+```
+
+## Safety Features
+
+### Production Safeguards
+- ⚠️ Explicit confirmation required for destructive operations
+- 🔒 Automatic backup before reset or rollback in production
+- ✅ Validation before applying migrations
+- 📝 Detailed logging of all operations
+- 🔄 Rollback capability for failed migrations
+
+### Transaction Safety
+- All migrations run within transactions
+- Automatic rollback on error
+- State consistency guaranteed
+
+### Backup Retention
+- Automatic backups retained for 30 days
+- Manual backups retained indefinitely
+- Cleanup command for old backups
+
+## Migration Best Practices
+
+### Creating Migrations
+
+1. **Use descriptive names**
+   ```
+   /db-operations migrate:create add_workflow_approvals
+   ```
+
+2. **Make migrations reversible**
+   Include both `up` and `down` migrations:
+   ```sql
+   -- Up
+   CREATE TABLE approvals (...);
+
+   -- Down
+   DROP TABLE approvals;
+   ```
+
+3. **Test migrations locally first**
+   ```
+   /db-operations migrate --env=dev
+   /db-operations migrate:rollback --env=dev
+   ```
+
+4. **Keep migrations small and focused**
+   - One logical change per migration
+   - Easier to review and rollback
+
+### Migration Dependencies
+
+Drizzle ORM ensures migrations run in order:
+```
+001_create_users_table.sql
+002_create_organizations_table.sql
+003_add_user_organization_relation.sql (depends on 001 and 002)
+```
+
+## Troubleshooting
+
+### Cannot Connect to Database
+```
+Error: Connection refused at localhost:5432
+```
+**Solution**: Ensure PostgreSQL is running
+```bash
+docker compose up -d postgres
+```
+
+### Migration Failed
+```
+Error: Migration 003_add_column.sql failed
+```
+**Solution**: Check migration SQL syntax and rollback
+```bash
+/db-operations migrate:rollback
+# Fix migration file
+/db-operations migrate
+```
+
+### Schema Mismatch
+```
+Error: Table 'users' does not match model definition
+```
+**Solution**: Run validation and create corrective migration
+```bash
+/db-operations validate
+/db-operations migrate:create fix_users_schema
+```
+
+### Seed Data Conflicts
+```
+Error: Duplicate key violation on users.email
+```
+**Solution**: Clear existing seed data first
+```bash
+/db-operations reset --force
+/db-operations seed
+```
+
+### Backup File Corrupted
+```
+Error: Invalid backup file format
+```
+**Solution**: Use a different backup or restore from automated backups
+```bash
+ls backups/db/automated/
+/db-operations restore backups/db/automated/latest.sql.gz
+```
+
+## CI/CD Integration
+
+### Azure DevOps Pipeline
+
+```yaml
+- task: Script@1
+  displayName: 'Database Migrations'
+  inputs:
+    script: |
+      /db-operations backup --env=$(ENVIRONMENT)
+      /db-operations migrate --env=$(ENVIRONMENT)
+      /db-operations validate --env=$(ENVIRONMENT)
+```
+
+### Pre-Deployment Checklist
+
+✅ Backup created
+✅ Migrations tested in staging
+✅ Schema validated
+✅ Rollback plan prepared
+
+## Drizzle ORM Schema Example
+
+```typescript
+// apps/backend/src/database/schema/users.schema.ts
+import { pgTable, serial, varchar, timestamp } from 'drizzle-orm/pg-core';
+
+export const users = pgTable('users', {
+  id: serial('id').primaryKey(),
+  email: varchar('email', { length: 255 }).notNull().unique(),
+  password: varchar('password', { length: 255 }).notNull(),
+  firstName: varchar('first_name', { length: 100 }),
+  lastName: varchar('last_name', { length: 100 }),
+  createdAt: timestamp('created_at').defaultNow(),
+  updatedAt: timestamp('updated_at').defaultNow(),
+});
+```
+
+## Related Commands
+
+- `/test-all backend` - Test database operations
+- `/deploy` - Deploy with database migrations
+- `/analyze-code` - Check migration file quality
+
+## Advanced Operations
+
+### Custom Seed Data
+
+Create custom seed files in `apps/backend/drizzle/seeds/`:
+
+```typescript
+// custom-seed.ts
+export async function seed(db: Database) {
+  await db.insert(users).values([
+    { email: 'custom@example.com', password: 'hashed' }
+  ]);
+}
+```
+
+### Partial Migrations
+
+Run specific migration range:
+```bash
+/db-operations migrate --from=001 --to=005
+```
+
+### Multi-Database Operations
+
+Operate on multiple databases:
+```bash
+/db-operations migrate --all-environments
+```
+
+---
+
+**Note**: Always test database operations in development before applying to staging or production. Keep backups and maintain a rollback plan for production changes.