gh-raw-labs-claude-code-marketplace-claude-plugin-plugins-mxcp-plugin/skills/mxcp-expert/references/comprehensive-testing-guide.md

# Comprehensive Testing Guide

**Complete testing strategy for MXCP servers: MXCP tests, Python unit tests, mocking, test databases, and concurrency safety.**

## Two Types of Tests

### 1. MXCP Tests (Integration Tests)

**Purpose**: Test the full tool/resource/prompt as it will be called by LLMs.

**Located**: In tool YAML files under `tests:` section

**Run with**: `mxcp test`

**Tests**:
- Tool can be invoked with parameters
- Return type matches specification
- Result structure is correct
- Parameter validation works

**Example**:
```yaml
# tools/get_customers.yml
mxcp: 1
tool:
  name: get_customers
  tests:
    - name: "basic_query"
      arguments:
        - key: city
          value: "Chicago"
      result:
        - customer_id: 3
          name: "Bob Johnson"
```

### 2. Python Unit Tests (Isolation Tests)

**Purpose**: Test Python functions in isolation with mocking, edge cases, concurrency.

**Located**: In `tests/` directory (pytest format)

**Run with**: `pytest` or `python -m pytest`

**Tests**:
- Function logic correctness
- Edge cases and error handling
- Mocked external dependencies
- Concurrency safety
- Result correctness verification

**Example**:
```python
# tests/test_api_wrapper.py
import pytest
from python.api_wrapper import fetch_users

@pytest.mark.asyncio
async def test_fetch_users_correctness():
    """Test that fetch_users returns correct structure"""
    result = await fetch_users(limit=5)

    assert "users" in result
    assert "count" in result
    assert result["count"] == 5
    assert len(result["users"]) == 5
    assert all("id" in user for user in result["users"])
```

## When to Use Which Tests

| Scenario | MXCP Tests | Python Unit Tests |
|----------|------------|-------------------|
| SQL-only tool | ✅ Required | ❌ Not applicable |
| Python tool (no external calls) | ✅ Required | ✅ Recommended |
| Python tool (with API calls) | ✅ Required | ✅ **Required** (with mocking) |
| Python tool (with DB writes) | ✅ Required | ✅ **Required** (test DB) |
| Python tool (async/concurrent) | ✅ Required | ✅ **Required** (concurrency tests) |

## Complete Testing Workflow

### Phase 1: MXCP Tests (Always First)

**For every tool, add test cases to YAML:**

```yaml
tool:
  name: my_tool
  # ... definition ...
  tests:
    - name: "happy_path"
      arguments:
        - key: param1
          value: "test_value"
      result:
        expected_field: "expected_value"

    - name: "edge_case_empty"
      arguments:
        - key: param1
          value: "nonexistent"
      result: []

    - name: "missing_optional_param"
      arguments: []
      # Should work with defaults
```

**Run**:
```bash
mxcp test tool my_tool
```

### Phase 2: Python Unit Tests (For Python Tools)

**Create test file structure**:
```bash
mkdir -p tests
touch tests/__init__.py
touch tests/test_my_module.py
```

**Write unit tests with pytest**:
```python
# tests/test_my_module.py
import pytest
from python.my_module import my_function

def test_my_function_correctness():
    """Verify correct results"""
    result = my_function("input")
    assert result["key"] == "expected_value"
    assert len(result["items"]) == 5

def test_my_function_edge_cases():
    """Test edge cases"""
    assert my_function("") == {"error": "Empty input"}
    assert my_function(None) == {"error": "Invalid input"}
```

**Run**:
```bash
pytest tests/
# Or with coverage
pytest --cov=python tests/
```

## Testing SQL Tools with Test Database

**CRITICAL**: SQL tools must be tested with real data to verify result correctness.

### Pattern 1: Use dbt Seeds for Test Data

```bash
# 1. Create test data seed
cat > seeds/test_data.csv <<'EOF'
id,name,value
1,test1,100
2,test2,200
3,test3,300
EOF

# 2. Create schema
cat > seeds/schema.yml <<'EOF'
version: 2
seeds:
  - name: test_data
    columns:
      - name: id
        tests: [unique, not_null]
EOF

# 3. Load test data
dbt seed --select test_data

# 4. Create tool with tests
cat > tools/query_test_data.yml <<'EOF'
mxcp: 1
tool:
  name: query_test_data
  parameters:
    - name: min_value
      type: integer
  return:
    type: array
  source:
    code: |
      SELECT * FROM test_data WHERE value >= $min_value
  tests:
    - name: "filter_200"
      arguments:
        - key: min_value
          value: 200
      result:
        - id: 2
          value: 200
        - id: 3
          value: 300
EOF

# 5. Test
mxcp test tool query_test_data
```

### Pattern 2: Create Test Fixtures in SQL

```sql
-- models/test_fixtures.sql
{{ config(materialized='table') }}

-- Create predictable test data
SELECT 1 as id, 'Alice' as name, 100 as score
UNION ALL
SELECT 2 as id, 'Bob' as name, 200 as score
UNION ALL
SELECT 3 as id, 'Charlie' as name, 150 as score
```

```yaml
# tools/top_scores.yml
tool:
  name: top_scores
  source:
    code: |
      SELECT * FROM test_fixtures ORDER BY score DESC LIMIT $limit
  tests:
    - name: "top_2"
      arguments:
        - key: limit
          value: 2
      result:
        - id: 2
          name: "Bob"
          score: 200
        - id: 3
          name: "Charlie"
          score: 150
```

### Pattern 3: Verify Aggregation Correctness

```yaml
# tools/calculate_stats.yml
tool:
  name: calculate_stats
  source:
    code: |
      SELECT
        COUNT(*) as total_count,
        SUM(score) as total_score,
        AVG(score) as avg_score,
        MAX(score) as max_score
      FROM test_fixtures
  tests:
    - name: "verify_aggregations"
      arguments: []
      result:
        - total_count: 3
          total_score: 450
          avg_score: 150.0
          max_score: 200
```

**If aggregations don't match expected values, the SQL logic is WRONG.**

## Testing Python Tools with Mocking

**CRITICAL**: Python tools with external API calls MUST use mocking in tests.

### Pattern 1: Mock HTTP Calls with pytest-httpx

```bash
# Install
pip install pytest-httpx
```

```python
# python/api_client.py
import httpx

async def fetch_external_data(api_key: str, user_id: int) -> dict:
    """Fetch data from external API"""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://api.example.com/users/{user_id}",
            headers={"Authorization": f"Bearer {api_key}"}
        )
        response.raise_for_status()
        return response.json()
```

```python
# tests/test_api_client.py
import pytest
from httpx import Response
from python.api_client import fetch_external_data

@pytest.mark.asyncio
async def test_fetch_external_data_success(httpx_mock):
    """Test successful API call with mocked response"""
    # Mock the HTTP call
    httpx_mock.add_response(
        url="https://api.example.com/users/123",
        json={"id": 123, "name": "Test User", "email": "test@example.com"}
    )

    # Call function
    result = await fetch_external_data("fake_api_key", 123)

    # Verify correctness
    assert result["id"] == 123
    assert result["name"] == "Test User"
    assert result["email"] == "test@example.com"

@pytest.mark.asyncio
async def test_fetch_external_data_error(httpx_mock):
    """Test API error handling"""
    httpx_mock.add_response(
        url="https://api.example.com/users/999",
        status_code=404,
        json={"error": "User not found"}
    )

    # Should handle error gracefully
    with pytest.raises(httpx.HTTPStatusError):
        await fetch_external_data("fake_api_key", 999)
```

### Pattern 2: Mock Database Calls

```python
# python/db_operations.py
from mxcp.runtime import db

def get_user_orders(user_id: int) -> list[dict]:
    """Get orders for a user"""
    result = db.execute(
        "SELECT * FROM orders WHERE user_id = $1",
        {"user_id": user_id}
    )
    return result.fetchall()
```

```python
# tests/test_db_operations.py
import pytest
from unittest.mock import Mock, MagicMock
from python.db_operations import get_user_orders

def test_get_user_orders(monkeypatch):
    """Test with mocked database"""
    # Create mock result
    mock_result = MagicMock()
    mock_result.fetchall.return_value = [
        {"order_id": 1, "user_id": 123, "amount": 50.0},
        {"order_id": 2, "user_id": 123, "amount": 75.0}
    ]

    # Mock db.execute
    mock_db = Mock()
    mock_db.execute.return_value = mock_result

    # Inject mock
    import python.db_operations
    monkeypatch.setattr(python.db_operations, "db", mock_db)

    # Test
    orders = get_user_orders(123)

    # Verify
    assert len(orders) == 2
    assert orders[0]["order_id"] == 1
    assert sum(o["amount"] for o in orders) == 125.0
```

### Pattern 3: Mock Third-Party Libraries

```python
# python/stripe_wrapper.py
import stripe

def create_customer(email: str, name: str) -> dict:
    """Create Stripe customer"""
    customer = stripe.Customer.create(email=email, name=name)
    return {"id": customer.id, "email": customer.email}
```

```python
# tests/test_stripe_wrapper.py
import pytest
from unittest.mock import Mock, patch
from python.stripe_wrapper import create_customer

@patch('stripe.Customer.create')
def test_create_customer(mock_create):
    """Test Stripe customer creation with mock"""
    # Mock Stripe response
    mock_customer = Mock()
    mock_customer.id = "cus_test123"
    mock_customer.email = "test@example.com"
    mock_create.return_value = mock_customer

    # Call function
    result = create_customer("test@example.com", "Test User")

    # Verify correctness
    assert result["id"] == "cus_test123"
    assert result["email"] == "test@example.com"

    # Verify Stripe was called correctly
    mock_create.assert_called_once_with(
        email="test@example.com",
        name="Test User"
    )
```

## Result Correctness Verification

**CRITICAL**: Tests must verify results are CORRECT, not just that code doesn't crash.

### Bad Test (Only checks structure):
```python
def test_calculate_total_bad():
    result = calculate_total([10, 20, 30])
    assert "total" in result  # ❌ Doesn't verify correctness
```

### Good Test (Verifies correct value):
```python
def test_calculate_total_good():
    result = calculate_total([10, 20, 30])
    assert result["total"] == 60  # ✅ Verifies correct calculation
    assert result["count"] == 3   # ✅ Verifies correct count
    assert result["average"] == 20.0  # ✅ Verifies correct average
```

### Pattern: Test Edge Cases for Correctness

```python
def test_aggregation_correctness():
    """Test various aggregations for correctness"""
    data = [
        {"id": 1, "value": 100},
        {"id": 2, "value": 200},
        {"id": 3, "value": 150}
    ]

    result = aggregate_data(data)

    # Verify each aggregation
    assert result["sum"] == 450  # 100 + 200 + 150
    assert result["avg"] == 150.0  # 450 / 3
    assert result["min"] == 100
    assert result["max"] == 200
    assert result["count"] == 3

    # Verify derived values
    assert result["range"] == 100  # 200 - 100
    assert result["median"] == 150

def test_empty_data_correctness():
    """Test edge case: empty data"""
    result = aggregate_data([])

    assert result["sum"] == 0
    assert result["avg"] == 0.0
    assert result["count"] == 0
    # Ensure no crashes, correct behavior for empty data
```

## Concurrency Safety for Python Tools

**CRITICAL**: MXCP tools run as a server - multiple requests can happen simultaneously.

### Common Concurrency Issues

#### ❌ WRONG: Global State with Race Conditions

```python
# python/unsafe_counter.py
counter = 0  # ❌ DANGER: Race condition!

def increment_counter() -> dict:
    global counter
    counter += 1  # ❌ Not thread-safe!
    return {"count": counter}

# Two simultaneous requests could both read counter=5,
# both increment to 6, both write 6 -> one increment lost!
```

#### ✅ CORRECT: Use Thread-Safe Approaches

**Option 1: Avoid shared state (stateless)**
```python
# python/safe_stateless.py
def process_request(data: dict) -> dict:
    """Completely stateless - safe for concurrent calls"""
    result = compute_something(data)
    return {"result": result}
    # No global state, no problem!
```

**Option 2: Use thread-safe structures**
```python
# python/safe_with_lock.py
import threading

counter_lock = threading.Lock()
counter = 0

def increment_counter() -> dict:
    global counter
    with counter_lock:  # ✅ Thread-safe
        counter += 1
        current = counter
    return {"count": current}
```

**Option 3: Use atomic operations**
```python
# python/safe_atomic.py
from threading import Lock
from collections import defaultdict

# Thread-safe counter
class SafeCounter:
    def __init__(self):
        self._value = 0
        self._lock = Lock()

    def increment(self):
        with self._lock:
            self._value += 1
            return self._value

counter = SafeCounter()

def increment_counter() -> dict:
    return {"count": counter.increment()}
```

### Concurrency-Safe Patterns

#### Pattern 1: Database as State (DuckDB is thread-safe)

```python
# python/db_counter.py
from mxcp.runtime import db

def increment_counter() -> dict:
    """Use database for state - thread-safe"""
    db.execute("""
        CREATE TABLE IF NOT EXISTS counter (
            id INTEGER PRIMARY KEY,
            value INTEGER
        )
    """)

    db.execute("""
        INSERT INTO counter (id, value) VALUES (1, 1)
        ON CONFLICT(id) DO UPDATE SET value = value + 1
    """)

    result = db.execute("SELECT value FROM counter WHERE id = 1")
    return {"count": result.fetchone()["value"]}
```

#### Pattern 2: Local Variables Only (Immutable)

```python
# python/safe_processing.py
async def process_data(input_data: list[dict]) -> dict:
    """Local variables only - safe for concurrent calls"""
    # All state is local to this function call
    results = []
    total = 0

    for item in input_data:
        processed = transform(item)  # Pure function
        results.append(processed)
        total += processed["value"]

    return {
        "results": results,
        "total": total,
        "count": len(results)
    }
    # When function returns, all state is discarded
```

#### Pattern 3: Async/Await (Concurrent, Not Parallel)

```python
# python/safe_async.py
import asyncio
import httpx

async def fetch_multiple_users(user_ids: list[int]) -> list[dict]:
    """Concurrent API calls - safe with async"""

    async def fetch_one(user_id: int) -> dict:
        # Each call has its own context - no shared state
        async with httpx.AsyncClient() as client:
            response = await client.get(f"https://api.example.com/users/{user_id}")
            return response.json()

    # Run concurrently, but each fetch_one is independent
    results = await asyncio.gather(*[fetch_one(uid) for uid in user_ids])
    return results
```

### Testing Concurrency Safety

```python
# tests/test_concurrency.py
import pytest
import asyncio
from python.my_module import concurrent_function

@pytest.mark.asyncio
async def test_concurrent_calls_no_race_condition():
    """Test that concurrent calls don't have race conditions"""

    # Run function 100 times concurrently
    tasks = [concurrent_function(i) for i in range(100)]
    results = await asyncio.gather(*tasks)

    # Verify all calls succeeded
    assert len(results) == 100

    # Verify no data corruption
    assert all(isinstance(r, dict) for r in results)

    # If function has a counter, verify correctness
    # (e.g., if each call increments, final count should be 100)

def test_parallel_execution_thread_safe():
    """Test with actual threading"""
    import threading

    results = []
    errors = []

    def worker(n):
        try:
            result = my_function(n)
            results.append(result)
        except Exception as e:
            errors.append(e)

    # Create 50 threads
    threads = [threading.Thread(target=worker, args=(i,)) for i in range(50)]

    # Start all threads
    for t in threads:
        t.start()

    # Wait for completion
    for t in threads:
        t.join()

    # Verify
    assert len(errors) == 0, f"Errors occurred: {errors}"
    assert len(results) == 50
```

## Complete Testing Checklist

### For SQL Tools:

- [ ] MXCP test cases in YAML
- [ ] Test with real seed data
- [ ] Verify result correctness (exact values)
- [ ] Test edge cases (empty results, NULL values)
- [ ] Test filters work correctly
- [ ] Test aggregations are mathematically correct
- [ ] Test with dbt test for data quality

### For Python Tools (No External Calls):

- [ ] MXCP test cases in YAML
- [ ] Python unit tests (pytest)
- [ ] Verify result correctness
- [ ] Test edge cases (empty input, NULL, invalid)
- [ ] Test error handling
- [ ] Test concurrency safety (if using shared state)

### For Python Tools (With External API Calls):

- [ ] MXCP test cases in YAML
- [ ] Python unit tests with mocking (pytest + httpx_mock)
- [ ] Mock all external API calls
- [ ] Test success path with mocked responses
- [ ] Test error cases (404, 500, timeout)
- [ ] Verify correct API parameters
- [ ] Test result correctness
- [ ] Test concurrency (multiple simultaneous calls)

### For Python Tools (With Database Operations):

- [ ] MXCP test cases in YAML
- [ ] Python unit tests
- [ ] Use test fixtures/seed data
- [ ] Verify query results correctness
- [ ] Test transactions (if applicable)
- [ ] Test concurrency (DuckDB is thread-safe)
- [ ] Clean up test data after tests

## Project Structure for Testing

```
project/
├── mxcp-site.yml
├── tools/
│   └── my_tool.yml              # Contains MXCP tests
├── python/
│   └── my_module.py             # Python code
├── tests/
│   ├── __init__.py
│   ├── test_my_module.py        # Python unit tests
│   ├── conftest.py              # pytest fixtures
│   └── fixtures/
│       └── test_data.json       # Test data
├── seeds/
│   ├── test_data.csv            # Test database seeds
│   └── schema.yml
└── requirements.txt             # Include: pytest, pytest-asyncio, pytest-httpx, pytest-cov
```

## Running Tests

```bash
# 1. MXCP tests (always run first)
mxcp validate  # Structure validation
mxcp test      # Integration tests

# 2. dbt tests (if using dbt)
dbt test

# 3. Python unit tests
pytest tests/ -v

# 4. With coverage report
pytest tests/ --cov=python --cov-report=html

# 5. Concurrency stress test (custom)
pytest tests/test_concurrency.py -v --count=100

# All together
mxcp validate && mxcp test && dbt test && pytest tests/ -v
```

## Summary

**Both types of tests are required**:

1. **MXCP tests** - Verify tools work end-to-end
2. **Python unit tests** - Verify logic, mocking, correctness, concurrency

**Key principles**:
- ✅ **Mock all external calls** - Use pytest-httpx, unittest.mock
- ✅ **Verify result correctness** - Don't just check structure
- ✅ **Use test databases** - SQL tools need real data
- ✅ **Test concurrency** - Tools run as servers
- ✅ **Avoid global mutable state** - Use stateless patterns or locks
- ✅ **Test edge cases** - Empty data, NULL, invalid input

**Before declaring a project done, BOTH test types must pass completely.**