zhongwei/gh-raw-labs-claude-code-marketplace-claude-plugin-plugins-mxcp-plugin

Fork 0

Files

Zhongwei Li adc4b2be25 Initial commit

2025-11-30 08:49:50 +08:00

24 KiB

Raw Blame History

Build and Validate Workflow

Mandatory workflow to ensure MXCP servers always work correctly.

Definition of Done

An MXCP server is DONE only when ALL of these criteria are met:

Virtual environment created: uv venv completed (if Python tools exist)
Dependencies installed: uv pip install mxcp black pyright pytest pytest-asyncio pytest-httpx pytest-cov (if Python tools exist)
Structure valid: mxcp validate passes with no errors
MXCP tests pass: mxcp test passes for all tools
Python code formatted: black python/ passes (if Python tools exist)
Type checking passes: pyright python/ passes with 0 errors (if Python tools exist)
Python unit tests pass: pytest tests/ -v passes (if Python tools exist)
Data quality: dbt test passes (if using dbt)
Result correctness verified: Tests check actual values, not just structure
Mocking implemented: External API calls are mocked in unit tests
Concurrency safe: Python tools avoid race conditions
Documentation quality verified: LLMs can understand tools with zero context
Error handling implemented: Python tools return structured errors
Manual verification: At least one manual test per tool succeeds
Security reviewed: Checklist completed (see below)
Config provided: Project has config.yml with usage instructions
Dependencies listed: requirements.txt includes all dev dependencies

NEVER declare a project complete without ALL checkboxes checked.

Mandatory Build Order

Follow this exact order to ensure correctness:

Phase 1: Foundation (Must complete before Phase 2)

Initialize project

mkdir project-name && cd project-name
mxcp init --bootstrap

Set up Python virtual environment (CRITICAL - do this BEFORE any MXCP commands)

# Create virtual environment with uv
uv venv

# Activate virtual environment
source .venv/bin/activate  # On Unix/macOS
# OR
.venv\Scripts\activate     # On Windows

# Verify activation (prompt should show (.venv))
which python
# Output: /path/to/project-name/.venv/bin/python

# Install MXCP and development tools
uv pip install mxcp black pyright pytest pytest-asyncio pytest-httpx pytest-cov

# Create requirements.txt for reproducibility
cat > requirements.txt <<'EOF'

mxcp>=0.1.0 black>=24.0.0 pyright>=1.1.0 pytest>=7.0.0 pytest-asyncio>=0.21.0 pytest-httpx>=0.21.0 pytest-cov>=4.0.0 EOF


**IMPORTANT**: Virtual environment must be active for ALL subsequent commands. If you close your terminal, re-activate with `source .venv/bin/activate`.

3. **Create project structure**
```bash
mkdir -p seeds models tools resources prompts python tests
touch tests/__init__.py

Set up dbt (if needed)

# Create dbt_project.yml if needed
# Create profiles.yml connection

Validation checkpoint: Verify structure

# Ensure virtual environment is active
echo $VIRTUAL_ENV  # Should show: /path/to/project-name/.venv

ls -la  # Confirm directories exist
mxcp validate  # Should pass (no tools yet, but structure valid)

CRITICAL: Directory Structure Enforcement

MXCP enforces organized directory structure. Files in wrong directories are ignored by discovery commands:

✅ Tools MUST be in tools/*.yml
✅ Resources MUST be in resources/*.yml
✅ Prompts MUST be in prompts/*.yml
❌ Tool files in root directory will be ignored
❌ Tool files in wrong directories will be ignored

Common mistake to avoid:

# ❌ WRONG - tool in root directory (will be ignored)
my_tool.yml

# ✅ CORRECT - tool in tools/ directory
tools/my_tool.yml

Use mxcp init --bootstrap to create proper structure automatically.

Phase 2: Data Layer (if applicable)

Add data source (CSV, Excel, etc.)

# Option A: CSV seed
cp data.csv seeds/

# Option B: Excel conversion
python -c "import pandas as pd; pd.read_excel('data.xlsx').to_csv('seeds/data.csv', index=False)"

Create schema.yml (CRITICAL - don't skip!)

# seeds/schema.yml
version: 2
seeds:
  - name: data
    description: "Data description here"
    columns:
      - name: id
        tests: [unique, not_null]
      # Add ALL columns with tests

Load and test data

dbt seed --select data
dbt test --select data

Validation checkpoint: Data quality verified

# Check data loaded
mxcp query "SELECT COUNT(*) FROM data"
# Should return row count

Phase 3: Build Tools ONE AT A TIME

CRITICAL: Build ONE tool, validate, test, THEN move to next.

For EACH tool:

Step 1: Create Test FIRST (with LLM-friendly documentation)

# tools/my_tool.yml
mxcp: 1
tool:
  name: my_tool
  description: "Retrieve data from table by filtering on column. Returns array of matching records. Use this to query specific records by their identifier."
  parameters:
    - name: param1
      type: string
      description: "Filter value for column (e.g., 'value123'). Must match exact column value."
      required: true
      examples: ["value123", "test_value"]
  return:
    type: array
    description: "Array of matching records"
    items:
      type: object
      properties:
        id: { type: integer, description: "Record identifier" }
        column: { type: string, description: "Filtered column value" }
  source:
    code: |
      SELECT * FROM data WHERE column = $param1
  tests:
    - name: "basic_test"
      arguments:
        - key: param1
          value: "test_value"
      result:
        # Expected result structure with actual values to verify
        - id: 1
          column: "test_value"

Documentation requirements (check before proceeding):

Tool description explains WHAT, returns WHAT, WHEN to use
Parameters have descriptions with examples
Return type properties all described
An LLM with zero context could understand how to use this

Step 2: Validate Structure

mxcp validate
# Must pass before proceeding

Common errors at this stage:

Indentation wrong (use spaces, not tabs)
Missing required fields (name, description, return)
Type mismatch (array vs object)
Invalid SQL syntax

If validation fails:

Read error message carefully
Check YAML indentation (use yamllint)
Verify all required fields present
Check type definitions match return data
Fix and re-validate

Step 3: Test Functionality

A. MXCP Integration Tests

# Run the test case
mxcp test tool my_tool

# Run manually with different inputs
mxcp run tool my_tool --param param1=test_value

If test fails:

Check SQL syntax in source
Verify table/column names exist
Test SQL directly: mxcp query "SELECT ..."
Check parameter binding ($param1 syntax)
Verify return type matches actual data
Fix and re-test

B. Python Code Quality (For Python Tools)

MANDATORY workflow after creating or editing ANY Python file:

# CRITICAL: Always ensure virtual environment is active first
source .venv/bin/activate

# Step 1: Format code with black
black python/
# Must see: "All done! ✨ 🍰 ✨" or "N file(s) reformatted"

# Step 2: Type check with pyright
pyright python/
# Must see: "0 errors, 0 warnings, 0 informations"

# Step 3: Run unit tests
pytest tests/ -v
# Must see: All tests PASSED

# If ANY step fails, fix before proceeding!

Create Unit Tests:

# Create test file
cat > tests/test_my_tool.py <<'EOF'
"""Tests for my_module."""

import pytest
from python.my_module import my_function
from typing import Dict, Any

def test_my_function_correctness():
    """Verify result correctness"""
    result = my_function("test_input")
    assert result["expected_key"] == "expected_value"  # Verify actual value!

@pytest.mark.asyncio
async def test_async_function():
    """Test async functions"""
    result = await async_function()
    assert result is not None
EOF

# Run tests with coverage
pytest tests/ -v --cov=python --cov-report=term-missing

Common Python Type Errors and Fixes:

# ❌ WRONG: Using 'any' type
from typing import Dict
async def get_data(id: str) -> Dict[str, any]:  # 'any' is not valid
    pass

# ✅ CORRECT: Use proper types
from typing import Dict, Any, Union
async def get_data(id: str) -> Dict[str, Union[str, int, float, bool]]:
    pass

# ✅ BETTER: Define response type
from typing import TypedDict
class DataResponse(TypedDict):
    success: bool
    data: str
    count: int

async def get_data(id: str) -> DataResponse:
    pass

If unit tests fail:

Check function logic
Verify test assertions are correct
Check imports
Fix and re-test

C. Mocking External Calls (Required for API tools)

# tests/test_api_tool.py
import pytest
from python.api_wrapper import fetch_data

@pytest.mark.asyncio
async def test_fetch_data_with_mock(httpx_mock):
    """Mock external API call"""
    # Mock the HTTP response
    httpx_mock.add_response(
        url="https://api.example.com/data",
        json={"key": "value", "count": 5}
    )

    # Call function
    result = await fetch_data("param")

    # Verify correctness
    assert result["key"] == "value"
    assert result["count"] == 5

D. Error Handling (Required for Python tools)

Python tools MUST return structured error objects, never raise exceptions to MXCP.

# python/my_module.py
import httpx

async def fetch_user(user_id: int) -> dict:
    """
    Fetch user with comprehensive error handling.

    Returns:
        Success: {"success": True, "user": {...}}
        Error: {"success": False, "error": "...", "error_code": "..."}
    """
    try:
        async with httpx.AsyncClient(timeout=10.0) as client:
            response = await client.get(
                f"https://api.example.com/users/{user_id}"
            )

            if response.status_code == 404:
                return {
                    "success": False,
                    "error": f"User with ID {user_id} not found. Use list_users to see available users.",
                    "error_code": "NOT_FOUND",
                    "user_id": user_id
                }

            if response.status_code >= 500:
                return {
                    "success": False,
                    "error": "External API is currently unavailable. Please try again later.",
                    "error_code": "API_ERROR",
                    "status_code": response.status_code
                }

            response.raise_for_status()

            return {
                "success": True,
                "user": response.json()
            }

    except httpx.TimeoutException:
        return {
            "success": False,
            "error": "Request timed out after 10 seconds. The API may be slow or unavailable.",
            "error_code": "TIMEOUT"
        }

    except Exception as e:
        return {
            "success": False,
            "error": f"Unexpected error: {str(e)}",
            "error_code": "UNKNOWN_ERROR"
        }

Test error handling:

# tests/test_error_handling.py
@pytest.mark.asyncio
async def test_user_not_found(httpx_mock):
    """Verify 404 returns structured error"""
    httpx_mock.add_response(
        url="https://api.example.com/users/999",
        status_code=404
    )

    result = await fetch_user(999)

    assert result["success"] is False
    assert result["error_code"] == "NOT_FOUND"
    assert "999" in result["error"]  # Error mentions the ID
    assert "list_users" in result["error"]  # Actionable suggestion

@pytest.mark.asyncio
async def test_timeout_error(httpx_mock):
    """Verify timeout returns structured error"""
    httpx_mock.add_exception(httpx.TimeoutException("Timeout"))

    result = await fetch_user(123)

    assert result["success"] is False
    assert result["error_code"] == "TIMEOUT"
    assert "timeout" in result["error"].lower()

Error message principles:

✅ Be specific (exactly what went wrong)
✅ Be actionable (suggest next steps)
✅ Provide context (relevant values/IDs)
✅ Use plain language (LLM-friendly)

See references/error-handling-guide.md for comprehensive patterns.

E. Concurrency Safety Tests (For stateful Python tools)

# tests/test_concurrency.py
import pytest
import asyncio

@pytest.mark.asyncio
async def test_concurrent_calls():
    """Verify no race conditions"""
    tasks = [my_function(i) for i in range(100)]
    results = await asyncio.gather(*tasks)

    # Verify all succeeded
    assert len(results) == 100
    assert all(r is not None for r in results)

Step 4: Verification Checkpoint

Before moving to next tool:

For ALL tools:

mxcp validate passes
mxcp test tool my_tool passes
Manual test with real data works
Tool returns expected data structure
Error cases handled (null params, no results, etc.)
Result correctness verified (not just structure)
Documentation quality verified:
- Tool description explains WHAT, WHAT it returns, WHEN to use
- All parameters have descriptions with examples
- Return fields all have descriptions
- Cross-references to related tools (if applicable)
LLM can understand with zero context (test: read YAML only, would you know how to use it?)

For Python tools (additionally):

Virtual environment active: echo $VIRTUAL_ENV shows path
Code formatted: black python/ shows "All done!"
Type checking passes: pyright python/ shows "0 errors"
pytest tests/test_my_tool.py -v passes
External calls are mocked (if applicable)
Concurrency tests pass (if stateful)
No global mutable state OR proper locking used
Test coverage >80% (pytest --cov=python tests/)
Error handling implemented:
- All try/except blocks return structured errors
- Error format: {"success": False, "error": "...", "error_code": "..."}
- Error messages are specific and actionable
- Never raise exceptions to MXCP (return error objects)

Only proceed to next tool when ALL checks pass.

Phase 4: Integration Testing

After all tools created:

Run full validation suite

# CRITICAL: Ensure virtual environment is active
source .venv/bin/activate

# Python code quality (if Python tools exist)
black python/                  # Must show: "All done!"
pyright python/                # Must show: "0 errors"
pytest tests/ -v --cov=python --cov-report=term  # All tests must pass

# MXCP validation and integration tests
mxcp validate  # All tools
mxcp test      # All tests
mxcp lint      # Documentation quality

# dbt tests (if applicable)
dbt test

Test realistic scenarios

# Test each tool with realistic inputs
mxcp run tool tool1 --param key=realistic_value
mxcp run tool tool2 --param key=realistic_value

# Test error cases
mxcp run tool tool1 --param key=invalid_value
mxcp run tool tool1  # Missing required param

Performance check (if applicable)

# Test with large inputs
mxcp run tool query_data --param limit=1000

# Check response time is reasonable
time mxcp run tool my_tool --param key=value

Phase 5: Security & Configuration

Security review checklist
- All SQL uses parameterized queries ($param)
- No hardcoded secrets in code
- Input validation on all parameters
- Sensitive fields filtered with policies (if needed)
- Authentication configured (if needed)

Create config.yml

# config.yml
mxcp: 1
profiles:
  default:
    secrets:
      - name: secret_name
        type: env
        parameters:
          env_var: SECRET_ENV_VAR

Create README or usage instructions

# Project Name

## Setup
1. Install dependencies: pip install -r requirements.txt
2. Set environment variables: export SECRET=xxx
3. Load data: dbt seed (if applicable)
4. Start server: mxcp serve

## Available Tools
- tool1: Description
- tool2: Description

Phase 6: Final Validation

This is the FINAL checklist before declaring DONE:

# 0. Activate virtual environment
source .venv/bin/activate
echo $VIRTUAL_ENV  # Must show path

# 1. Python code quality (if Python tools exist)
black python/ && pyright python/ && pytest tests/ -v
# All must pass

# 2. Clean start test
cd .. && cd project-name
mxcp validate
# Should pass

# 3. All tests pass
mxcp test
# Should show all tests passing

# 4. Manual smoke test
mxcp run tool <main_tool> --param key=value
# Should return valid data

# 5. Lint check
mxcp lint
# Should have no critical issues

# 6. dbt tests (if applicable)
dbt test
# All data quality tests pass

# 7. Serve test
mxcp serve --transport http --port 8080 &
SERVER_PID=$!
sleep 2
curl http://localhost:8080/health || true
kill $SERVER_PID
# Server should start without errors

Common Failure Patterns & Fixes

YAML Validation Errors

Error: "Invalid YAML: expected "

# WRONG: Mixed spaces and tabs
tool:
  name: my_tool
    description: "..."  # Tab here

# CORRECT: Consistent spaces (2 or 4)
tool:
  name: my_tool
  description: "..."

Error: "Missing required field: description"

# WRONG: Missing description
tool:
  name: my_tool
  parameters: [...]

# CORRECT: All required fields
tool:
  name: my_tool
  description: "What this tool does"
  parameters: [...]

Error: "Invalid type for field 'return'"

# WRONG: String instead of type object
return: "array"

# CORRECT: Proper type definition
return:
  type: array
  items:
    type: object

SQL Errors

Error: "Table 'xyz' not found"

-- WRONG: Table doesn't exist
SELECT * FROM xyz

-- FIX: Check table name, run dbt seed
SELECT * FROM actual_table_name

-- VERIFY: List tables
-- mxcp query "SHOW TABLES"

Error: "Column 'abc' not found"

-- WRONG: Column name typo or doesn't exist
SELECT abc FROM table

-- FIX: Check exact column name (case-sensitive in some DBs)
SELECT actual_column_name FROM table

-- VERIFY: List columns
-- mxcp query "DESCRIBE table"

Error: "Unbound parameter: $param1"

# WRONG: Parameter not defined in parameters list
parameters:
  - name: other_param
source:
  code: SELECT * FROM table WHERE col = $param1

# CORRECT: Define all parameters used in SQL
parameters:
  - name: param1
    type: string
source:
  code: SELECT * FROM table WHERE col = $param1

Type Mismatch Errors

Error: "Expected object, got array"

# WRONG: Return type doesn't match actual data
return:
  type: object
source:
  code: SELECT * FROM table  # Returns multiple rows (array)

# CORRECT: Match return type to SQL result
return:
  type: array
  items:
    type: object
source:
  code: SELECT * FROM table

Error: "Expected string, got number"

# WRONG: Parameter type doesn't match usage
parameters:
  - name: age
    type: string
source:
  code: SELECT * FROM users WHERE age > $age  # Numeric comparison

# CORRECT: Use appropriate type
parameters:
  - name: age
    type: integer
source:
  code: SELECT * FROM users WHERE age > $age

Python Import Errors

Error: "ModuleNotFoundError: No module named 'pandas'"

# WRONG: Library not installed OR virtual environment not active
import pandas as pd

# FIX:
# 1. Ensure virtual environment is active
source .venv/bin/activate

# 2. Add to requirements.txt
echo "pandas>=2.0.0" >> requirements.txt

# 3. Install using uv
uv pip install pandas

Error: "ImportError: cannot import name 'db' from 'mxcp.runtime'"

# WRONG: Import path incorrect
from mxcp import db

# CORRECT: Import from runtime
from mxcp.runtime import db

Python Code Quality Errors

Error: Black formatting fails with "INTERNAL ERROR"

# WRONG: Syntax error in Python code
# FIX: Check syntax first
python -m py_compile python/your_file.py
# Fix syntax errors, then run black
black python/

Error: Pyright shows "Type of 'any' is unknown"

# WRONG: Using lowercase 'any'
def get_data() -> Dict[str, any]:
    pass

# CORRECT: Use 'Any' from typing
from typing import Dict, Any
def get_data() -> Dict[str, Any]:
    pass

Error: "command not found: mxcp"

# WRONG: Virtual environment not active
mxcp validate

# FIX: Activate virtual environment
source .venv/bin/activate
which mxcp  # Should show: /path/to/project/.venv/bin/mxcp
mxcp validate

dbt Errors

Error: "Seed file not found"

# WRONG: File not in seeds/ directory
dbt seed --select data

# FIX: Check file location
ls seeds/
# Ensure data.csv exists in seeds/

# Or check seed name matches filename
# seeds/my_data.csv → dbt seed --select my_data

Error: "Test failed: unique_column_id"

# Data has duplicates
# FIX: Clean data or remove test
seeds:
  - name: data
    columns:
      - name: id
        tests: [unique]  # Remove if duplicates are valid

Debugging Workflow

When something doesn't work:

Step 1: Identify the Layer

YAML layer: mxcp validate fails → YAML structure issue
SQL layer: mxcp test fails but validate passes → SQL issue
Data layer: SQL syntax OK but wrong results → Data issue
Type layer: Runtime error about types → Type mismatch
Python layer: Import or runtime error → Python code issue

Step 2: Isolate the Problem

# Test YAML structure
mxcp validate --debug

# Test SQL directly
mxcp query "SELECT * FROM table LIMIT 5"

# Test tool with minimal input
mxcp run tool my_tool --param key=simple_value

# Check logs
mxcp serve --debug
# Look for error messages

Step 3: Fix Incrementally

Fix one error at a time
Re-validate after each fix
Don't move forward until green

Step 4: Verify Fix

# After fixing, run full suite
mxcp validate && mxcp test && mxcp lint

# If all pass, manual test
mxcp run tool my_tool --param key=test_value

Self-Checking for Agents

Before declaring a project complete, agent must verify:

0. Is virtual environment set up? (CRITICAL)

# Check virtual environment exists
ls .venv/bin/activate  # Must exist

# Activate it
source .venv/bin/activate

# Verify activation
echo $VIRTUAL_ENV  # Must show: /path/to/project/.venv
which python  # Must show: /path/to/project/.venv/bin/python

1. Can project be initialized?

cd project-directory
ls mxcp-site.yml  # Must exist

2. Python code quality passes? (if Python tools exist)

# Ensure venv active first
source .venv/bin/activate

# Check formatting
black --check python/
# Exit code 0 = success

# Check types
pyright python/
# Must show: "0 errors, 0 warnings, 0 informations"

# Check tests
pytest tests/ -v
# All tests show PASSED

3. Does MXCP validation pass?

# Ensure venv active
source .venv/bin/activate

mxcp validate
# Exit code 0 = success

4. Do MXCP tests pass?

# Ensure venv active
source .venv/bin/activate

mxcp test
# All tests show PASSED

5. Can tools be executed?

# Ensure venv active
source .venv/bin/activate

mxcp run tool <each_tool> --param key=value
# Returns data without errors

6. Is configuration complete?

ls config.yml  # Exists
cat config.yml | grep "mxcp: 1"  # Valid

7. Are dependencies listed?

# Must have requirements.txt with all dependencies
ls requirements.txt  # Exists
cat requirements.txt  # Has mxcp, black, pyright, pytest

8. Can server start?

# Ensure venv active
source .venv/bin/activate

timeout 5 mxcp serve --transport http --port 8080 || true
# Should start without immediate errors

Retry Strategy

If validation fails:

Attempt 1: Fix Based on Error Message

Read error message carefully
Apply specific fix
Re-validate

Attempt 2: Check Examples

Compare with working examples
Verify structure matches pattern
Re-validate

Attempt 3: Simplify

Remove optional features
Test minimal version
Add features back incrementally

If Still Failing:

Report exact error to user
Provide working minimal example
Ask for clarification on requirements

Summary: The Golden Rule

Build → Validate → Test → Verify → THEN Next

Never skip steps. Never batch multiple tools without validating each one. Always verify before declaring done.

If validation fails, the project is NOT done. Fix until all checks pass.

24 KiB Raw Blame History