Initial commit

2025-11-30 08:59:43 +08:00
commit 966ef521f7
25 changed files with 9763 additions and 0 deletions
--- a/skills/test-data-management/SKILL.md
+++ b/skills/test-data-management/SKILL.md
@@ -0,0 +1,419 @@
+---
+name: test-data-management
+description: Use when fixing flaky tests from data pollution, choosing between fixtures and factories, setting up test data isolation, handling PII in tests, or seeding test databases - provides isolation strategies and anti-patterns
+---
+
+# Test Data Management
+
+## Overview
+
+**Core principle:** Test isolation first. Each test should work independently regardless of execution order.
+
+**Rule:** Never use production data in tests without anonymization.
+
+## Test Isolation Decision Tree
+
+| Symptom | Root Cause | Solution |
+|---------|------------|----------|
+| Tests pass alone, fail together | Shared database state | Use transactions with rollback |
+| Tests fail intermittently | Race conditions on shared data | Use unique IDs per test |
+| Tests leave data behind | No cleanup | Add explicit teardown fixtures |
+| Slow test setup/teardown | Creating too much data | Use factories, minimal data |
+| Can't reproduce failures | Non-deterministic data | Use fixtures with static data |
+
+**Primary strategy:** Database transactions (wrap test in transaction, rollback after). Fastest and most reliable.
+
+## Fixtures vs Factories Quick Guide
+
+| Use Fixtures (Static Files) | Use Factories (Code Generators) |
+|------------------------------|----------------------------------|
+| Integration/contract tests | Unit tests |
+| Realistic complex scenarios | Need many variations |
+| Specific edge cases to verify | Simple "valid object" needed |
+| Team needs to review data | Randomized/parameterized tests |
+| Data rarely changes | Frequent maintenance |
+
+**Decision:** Static, complex, reviewable → Fixtures. Dynamic, simple, variations → Factories.
+
+**Hybrid (recommended):** Fixtures for integration tests, factories for unit tests.
+
+## Anti-Patterns Catalog
+
+### ❌ Shared Test Data
+**Symptom:** All tests use same "test_user_123" in database
+
+**Why bad:** Tests pollute each other, fail when run in parallel, can't isolate failures
+
+**Fix:** Each test creates its own data with unique IDs or uses transactions
+
+---
+
+### ❌ No Cleanup Strategy
+**Symptom:** Database grows with every test run, tests fail on second run
+
+**Why bad:** Leftover data causes unique constraint violations, flaky tests
+
+**Fix:** Use transaction rollback or explicit teardown fixtures
+
+---
+
+### ❌ Production Data in Tests
+**Symptom:** Copying production database to test environment
+
+**Why bad:** Privacy violations (GDPR, CCPA), security risk, compliance issues
+
+**Fix:** Use synthetic data generation or anonymized/masked data
+
+---
+
+### ❌ Hardcoded Test Data
+**Symptom:** Every test creates `User(name="John", email="john@test.com")`
+
+**Why bad:** Violates DRY, maintenance nightmare when schema changes, no variations
+
+**Fix:** Use factories to generate test data programmatically
+
+---
+
+### ❌ Copy-Paste Fixtures
+**Symptom:** 50 nearly-identical JSON fixture files
+
+**Why bad:** Hard to maintain, changes require updating all copies
+
+**Fix:** Use fixture inheritance or factory-generated fixtures
+
+## Isolation Strategies Quick Reference
+
+| Strategy | Speed | Use When | Pros | Cons |
+|----------|-------|----------|------|------|
+| **Transactions (Rollback)** | Fast | Database tests | No cleanup code, bulletproof | DB only |
+| **Unique IDs (UUID/timestamp)** | Fast | Parallel tests, external APIs | No conflicts | Still needs cleanup |
+| **Explicit Cleanup (Teardown)** | Medium | Files, caches, APIs | Works for anything | Manual code |
+| **In-Memory Database** | Fastest | Unit tests | Complete isolation | Not production-like |
+| **Test Containers** | Medium | Integration tests | Production-like | Slower startup |
+
+**Recommended order:** Try transactions first, add unique IDs for parallelization, explicit cleanup as last resort.
+
+## Data Privacy Quick Guide
+
+| Data Type | Strategy | Why |
+|-----------|----------|-----|
+| **PII (names, emails, addresses)** | Synthetic generation (Faker) | Avoid legal risk |
+| **Payment data** | NEVER use production | PCI-DSS compliance |
+| **Health data** | Anonymize + subset | HIPAA compliance |
+| **Sensitive business data** | Mask or synthesize | Protect IP |
+| **Non-sensitive metadata** | Can use production | ID mappings, timestamps OK if no PII |
+
+**Default rule:** When in doubt, use synthetic data.
+
+## Your First Test Data Setup
+
+**Start minimal, add complexity only when needed:**
+
+**Phase 1: Transactions (Week 1)**
+```python
+@pytest.fixture
+def db_session(db_engine):
+    connection = db_engine.connect()
+    transaction = connection.begin()
+    session = Session(bind=connection)
+
+    yield session
+
+    transaction.rollback()
+    connection.close()
+```
+
+**Phase 2: Add Factories (Week 2)**
+```python
+class UserFactory:
+    @staticmethod
+    def create(**overrides):
+        defaults = {
+            "id": str(uuid4()),
+            "email": f"test_{uuid4()}@example.com",
+            "created_at": datetime.now()
+        }
+        return {**defaults, **overrides}
+```
+
+**Phase 3: Add Fixtures for Complex Cases (Week 3+)**
+```json
+// tests/fixtures/valid_invoice.json
+{
+  "id": "inv-001",
+  "items": [/* complex nested data */],
+  "total": 107.94
+}
+```
+
+**Don't start with full complexity.** Master transactions first.
+
+## Non-Database Resource Isolation
+
+Database transactions don't work for files, caches, message queues, or external services. Use **explicit cleanup with unique namespacing**.
+
+### Temporary Files Strategy
+
+**Recommended:** Python's `tempfile` module (automatic cleanup)
+
+```python
+import tempfile
+from pathlib import Path
+
+@pytest.fixture
+def temp_workspace():
+    """Isolated temporary directory for test"""
+    with tempfile.TemporaryDirectory(prefix="test_") as tmp_dir:
+        yield Path(tmp_dir)
+    # Automatic cleanup on exit
+```
+
+**Alternative (manual control):**
+```python
+from uuid import uuid4
+import shutil
+
+@pytest.fixture
+def temp_dir():
+    test_dir = Path(f"/tmp/test_{uuid4()}")
+    test_dir.mkdir(parents=True)
+
+    yield test_dir
+
+    shutil.rmtree(test_dir, ignore_errors=True)
+```
+
+### Redis/Cache Isolation Strategy
+
+**Option 1: Unique key namespace per test (lightweight)**
+
+```python
+@pytest.fixture
+def redis_namespace(redis_client):
+    """Namespaced Redis keys with automatic cleanup"""
+    namespace = f"test:{uuid4()}"
+
+    yield namespace
+
+    # Cleanup: Delete all keys with this namespace
+    for key in redis_client.scan_iter(f"{namespace}:*"):
+        redis_client.delete(key)
+
+def test_caching(redis_namespace, redis_client):
+    key = f"{redis_namespace}:user:123"
+    redis_client.set(key, "value")
+    # Automatic cleanup after test
+```
+
+**Option 2: Separate Redis database per test (stronger isolation)**
+
+```python
+@pytest.fixture
+def isolated_redis():
+    """Use Redis DB 1-15 for tests (DB 0 for dev)"""
+    import random
+    test_db = random.randint(1, 15)
+    client = redis.Redis(db=test_db)
+
+    yield client
+
+    client.flushdb()  # Clear entire test database
+```
+
+**Option 3: Test containers (best isolation, slower)**
+
+```python
+from testcontainers.redis import RedisContainer
+
+@pytest.fixture(scope="session")
+def redis_container():
+    with RedisContainer() as container:
+        yield container
+
+@pytest.fixture
+def redis_client(redis_container):
+    client = redis.from_url(redis_container.get_connection_url())
+    yield client
+    client.flushdb()
+```
+
+### Combined Resource Cleanup
+
+When tests use database + files + cache:
+
+```python
+@pytest.fixture
+def isolated_test_env(db_session, temp_workspace, redis_namespace):
+    """Combined isolation for all resources"""
+    yield {
+        "db": db_session,
+        "files": temp_workspace,
+        "cache_ns": redis_namespace
+    }
+    # Teardown automatic via dependent fixtures
+    # Order: External resources first, DB last
+```
+
+### Quick Decision Guide
+
+| Resource Type | Isolation Strategy | Cleanup Method |
+|---------------|-------------------|----------------|
+| **Temporary files** | Unique directory per test | `tempfile.TemporaryDirectory()` |
+| **Redis cache** | Unique key namespace | Delete by pattern in teardown |
+| **Message queues** | Unique queue name | Delete queue in teardown |
+| **External APIs** | Unique resource IDs | DELETE requests in teardown |
+| **Test containers** | Per-test container | Container auto-cleanup |
+
+**Rule:** If transactions don't work, use unique IDs + explicit cleanup.
+
+## Test Containers Pattern
+
+**Core principle:** Session-scoped container + transaction rollback per test.
+
+**Don't recreate containers per test** - startup overhead kills performance.
+
+### SQL Database Containers (PostgreSQL, MySQL)
+
+**Recommended:** Session-scoped container + transactional fixtures
+
+```python
+from testcontainers.postgres import PostgresContainer
+import pytest
+
+@pytest.fixture(scope="session")
+def postgres_container():
+    """Container lives for entire test run"""
+    with PostgresContainer("postgres:15") as container:
+        yield container
+        # Auto-cleanup after all tests
+
+@pytest.fixture
+def db_session(postgres_container):
+    """Transaction per test - fast isolation"""
+    engine = create_engine(postgres_container.get_connection_url())
+    connection = engine.connect()
+    transaction = connection.begin()
+    session = Session(bind=connection)
+
+    yield session
+
+    transaction.rollback()  # <1ms cleanup
+    connection.close()
+```
+
+**Performance:**
+- Container startup: 5-10 seconds (once per test run)
+- Transaction rollback: <1ms per test
+- 100 tests: ~10 seconds total vs 8-16 minutes if recreating container per test
+
+**When to recreate container:**
+- Testing database migrations (need clean schema each time)
+- Testing database extensions/configuration changes
+- Container state itself is under test
+
+**For data isolation:** Transactions within shared container always win.
+
+### NoSQL/Cache Containers (Redis, MongoDB)
+
+Use session-scoped container + flush per test:
+
+```python
+from testcontainers.redis import RedisContainer
+
+@pytest.fixture(scope="session")
+def redis_container():
+    """Container lives for entire test run"""
+    with RedisContainer() as container:
+        yield container
+
+@pytest.fixture
+def redis_client(redis_container):
+    """Fresh client per test"""
+    client = redis.from_url(redis_container.get_connection_url())
+    yield client
+    client.flushdb()  # Clear after test
+```
+
+### Container Scope Decision
+
+| Use Case | Container Scope | Data Isolation Strategy |
+|----------|-----------------|------------------------|
+| SQL database tests | `scope="session"` | Transaction rollback per test |
+| NoSQL cache tests | `scope="session"` | Flush database per test |
+| Migration testing | `scope="function"` | Fresh schema per test |
+| Service integration | `scope="session"` | Unique IDs + cleanup per test |
+
+**Default:** Session scope + transaction/flush per test (100x faster).
+
+## Common Mistakes
+
+### ❌ Creating Full Objects When Partial Works
+**Symptom:** Test needs user ID, creates full user with 20 fields
+
+**Fix:** Create minimal valid object:
+```python
+# ❌ Bad
+user = UserFactory.create(
+    name="Test", email="test@example.com",
+    address="123 St", phone="555-1234",
+    # ... 15 more fields
+)
+
+# ✅ Good
+user = {"id": str(uuid4())}  # If only ID needed
+```
+
+---
+
+### ❌ No Transaction Isolation for Database Tests
+**Symptom:** Writing manual cleanup code for every database test
+
+**Fix:** Use transactional fixtures. Wrap in transaction, automatic rollback.
+
+---
+
+### ❌ Testing With Timestamps That Fail at Midnight
+**Symptom:** Tests pass during day, fail at exactly midnight
+
+**Fix:** Mock system time or use relative dates:
+```python
+# ❌ Bad
+assert created_at.date() == datetime.now().date()
+
+# ✅ Good
+from freezegun import freeze_time
+@freeze_time("2025-11-15 12:00:00")
+def test_timestamp():
+    assert created_at.date() == date(2025, 11, 15)
+```
+
+## Quick Reference
+
+**Test Isolation Priority:**
+1. Database tests → Transactions (rollback)
+2. Parallel execution → Unique IDs (UUID)
+3. External services → Explicit cleanup
+4. Files/caches → Teardown fixtures
+
+**Fixtures vs Factories:**
+- Complex integration scenario → Fixture
+- Simple unit test → Factory
+- Need variations → Factory
+- Specific edge case → Fixture
+
+**Data Privacy:**
+- PII/sensitive → Synthetic data (Faker, custom generators)
+- Never production payment/health data
+- Mask if absolutely need production structure
+
+**Getting Started:**
+1. Add transaction fixtures (Week 1)
+2. Add factory for common objects (Week 2)
+3. Add complex fixtures as needed (Week 3+)
+
+## Bottom Line
+
+**Test isolation prevents flaky tests.**
+
+Use transactions for database tests (fastest, cleanest). Use factories for unit tests (flexible, DRY). Use fixtures for complex integration scenarios (realistic, reviewable). Never use production data without anonymization.