# Python Runtime API Reference

Complete reference for MXCP Python endpoints, including wrapping external libraries and packages.

## Database Access

```python
from mxcp.runtime import db

# Execute query
results = db.execute(
    "SELECT * FROM users WHERE id = $id",
    {"id": user_id}
)

# Get first result
first = results[0] if results else None

# Iterate results
for row in results:
    print(row["name"])
```

**Important**: Always access through `db.execute()`, never cache `db.connection`.

## Configuration & Secrets

```python
from mxcp.runtime import config

# Get secret (returns dict with parameters)
secret = config.get_secret("api_key")
api_key = secret["value"] if secret else None

# For complex secrets (like HTTP with headers)
http_secret = config.get_secret("api_service")
if http_secret:
    token = http_secret.get("BEARER_TOKEN")
    headers = http_secret.get("EXTRA_HTTP_HEADERS", {})

# Get settings
project_name = config.get_setting("project")
debug_mode = config.get_setting("debug", default=False)

# Access full configs
user_config = config.user_config
site_config = config.site_config
```

## Lifecycle Hooks

```python
from mxcp.runtime import on_init, on_shutdown
import httpx

client = None

@on_init
def setup():
    """Initialize resources at startup"""
    global client
    client = httpx.Client()
    print("Client initialized")

@on_shutdown
def cleanup():
    """Clean up resources at shutdown"""
    global client
    if client:
        client.close()
```

**IMPORTANT: Lifecycle hooks are for Python resources ONLY**

- ✅ **USE FOR**: HTTP clients, external API connections, ML model loading, cache clients
- ❌ **DON'T USE FOR**: Database management, DuckDB connections, dbt operations

The DuckDB connection is managed automatically by MXCP. These hooks are for managing Python-specific resources that need initialization at server startup and cleanup at shutdown.

## Async Functions

```python
import asyncio
import aiohttp

async def fetch_data(urls: list[str]) -> list[dict]:
    """Fetch from multiple URLs concurrently"""
    
    async def fetch_one(url: str) -> dict:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                return await response.json()
    
    results = await asyncio.gather(*[fetch_one(url) for url in urls])
    return results
```

## Return Types

Match your function return to the endpoint's return type:

```python
# Array return
def list_items() -> list:
    return [{"id": 1}, {"id": 2}]

# Object return
def get_stats() -> dict:
    return {"total": 100, "active": 75}

# Scalar return
def count_items() -> int:
    return 42
```

## Shared Modules

Organize code in subdirectories:

```python
# python/utils/validators.py
def validate_email(email: str) -> bool:
    import re
    return bool(re.match(r'^[\w\.-]+@[\w\.-]+\.\w+$', email))

# python/main_tool.py
from utils.validators import validate_email

def process_user(email: str) -> dict:
    if not validate_email(email):
        return {"error": "Invalid email"}
    return {"status": "ok"}
```

## Error Handling

```python
def safe_divide(a: float, b: float) -> dict:
    if b == 0:
        return {"error": "Division by zero"}
    return {"result": a / b}
```

## External API Integration Pattern

```python
import httpx
from mxcp.runtime import config, db

async def call_external_api(param: str) -> dict:
    # Get API key
    api_key = config.get_secret("external_api")["value"]
    
    # Check cache
    cached = db.execute(
        "SELECT data FROM cache WHERE key = $key AND ts > datetime('now', '-1 hour')",
        {"key": param}
    ).fetchone()
    
    if cached:
        return cached["data"]
    
    # Make API call
    async with httpx.AsyncClient() as client:
        response = await client.get(
            "https://api.example.com/data",
            params={"q": param, "key": api_key}
        )
        data = response.json()
    
    # Cache result
    db.execute(
        "INSERT OR REPLACE INTO cache (key, data, ts) VALUES ($1, $2, CURRENT_TIMESTAMP)",
        {"key": param, "data": data}
    )
    
    return data
```

## Database Reload (Advanced)

Use `reload_duckdb` only when external tools need exclusive database access:

```python
from mxcp.runtime import reload_duckdb

def rebuild_database():
    """Trigger database rebuild"""
    
    def rebuild():
        # Run with exclusive database access
        import subprocess
        subprocess.run(["dbt", "run"], check=True)
    
    reload_duckdb(
        payload_func=rebuild,
        description="Rebuilding with dbt"
    )
    
    return {"status": "Reload scheduled"}
```

**Note**: Normally you don't need this. Use `db.execute()` for direct operations.

## Wrapping External Libraries

### Pattern 1: Simple Library Wrapper

**Use case**: Expose existing Python library as MCP tool

```python
# python/library_wrapper.py
"""Wrapper for an existing library like requests, pandas, etc."""

import requests
from mxcp.runtime import get_secret

def fetch_url(url: str, method: str = "GET", headers: dict = None) -> dict:
    """Wrap requests library as MCP tool"""
    try:
        # Get auth if needed
        secret = get_secret("api_token")
        if secret and headers is None:
            headers = {"Authorization": f"Bearer {secret['token']}"}

        response = requests.request(method, url, headers=headers, timeout=30)
        response.raise_for_status()

        return {
            "status_code": response.status_code,
            "headers": dict(response.headers),
            "body": response.json() if response.headers.get('content-type', '').startswith('application/json') else response.text
        }
    except requests.RequestException as e:
        return {"error": str(e), "status": "failed"}
```

```yaml
# tools/http_request.yml
mxcp: 1
tool:
  name: http_request
  description: "Make HTTP requests using requests library"
  language: python
  parameters:
    - name: url
      type: string
    - name: method
      type: string
      default: "GET"
  return:
    type: object
  source:
    file: ../python/library_wrapper.py
```

### Pattern 2: Data Science Library Wrapper (pandas, numpy)

```python
# python/data_analysis.py
"""Wrap pandas for data analysis"""

import pandas as pd
import numpy as np
from mxcp.runtime import db

def analyze_dataframe(table_name: str) -> dict:
    """Analyze a table using pandas"""
    # Read from DuckDB into pandas
    df = db.execute(f"SELECT * FROM {table_name}").df()

    # Pandas analysis
    analysis = {
        "shape": df.shape,
        "columns": list(df.columns),
        "dtypes": df.dtypes.astype(str).to_dict(),
        "missing_values": df.isnull().sum().to_dict(),
        "summary_stats": df.describe().to_dict(),
        "memory_usage": df.memory_usage(deep=True).sum()
    }

    # Numeric column correlations
    numeric_cols = df.select_dtypes(include=[np.number]).columns
    if len(numeric_cols) > 1:
        analysis["correlations"] = df[numeric_cols].corr().to_dict()

    return analysis

def pandas_query(table_name: str, operation: str) -> dict:
    """Execute pandas operations on DuckDB table"""
    df = db.execute(f"SELECT * FROM {table_name}").df()

    # Support common pandas operations
    if operation == "describe":
        result = df.describe().to_dict()
    elif operation == "head":
        result = df.head(10).to_dict('records')
    elif operation == "value_counts":
        # For first categorical column
        cat_col = df.select_dtypes(include=['object']).columns[0]
        result = df[cat_col].value_counts().to_dict()
    else:
        return {"error": f"Unknown operation: {operation}"}

    return {"operation": operation, "result": result}
```

### Pattern 3: ML Library Wrapper (scikit-learn)

```python
# python/ml_wrapper.py
"""Wrap scikit-learn for ML tasks"""

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from mxcp.runtime import db, on_init
import pickle
import os

# Global model store
models = {}

@on_init
def load_models():
    """Load saved models on startup"""
    global models
    model_dir = "models"
    if os.path.exists(model_dir):
        for file in os.listdir(model_dir):
            if file.endswith('.pkl'):
                model_name = file[:-4]
                with open(os.path.join(model_dir, file), 'rb') as f:
                    models[model_name] = pickle.load(f)

def train_classifier(
    table_name: str,
    target_column: str,
    feature_columns: list[str],
    model_name: str = "default"
) -> dict:
    """Train a classifier on DuckDB table"""
    # Load data
    df = db.execute(f"SELECT * FROM {table_name}").df()

    X = df[feature_columns]
    y = df[target_column]

    # Split data
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    # Train model
    model = RandomForestClassifier(n_estimators=100, random_state=42)
    model.fit(X_train, y_train)

    # Evaluate
    train_score = model.score(X_train, y_train)
    test_score = model.score(X_test, y_test)

    # Save model
    global models
    models[model_name] = model
    os.makedirs("models", exist_ok=True)
    with open(f"models/{model_name}.pkl", 'wb') as f:
        pickle.dump(model, f)

    return {
        "model_name": model_name,
        "train_accuracy": train_score,
        "test_accuracy": test_score,
        "feature_importance": dict(zip(feature_columns, model.feature_importances_))
    }

def predict(model_name: str, features: dict) -> dict:
    """Make prediction with trained model"""
    if model_name not in models:
        return {"error": f"Model '{model_name}' not found"}

    model = models[model_name]

    # Convert features to DataFrame with correct order
    import pandas as pd
    feature_df = pd.DataFrame([features])

    prediction = model.predict(feature_df)[0]
    probabilities = model.predict_proba(feature_df)[0] if hasattr(model, 'predict_proba') else None

    return {
        "prediction": prediction,
        "probabilities": probabilities.tolist() if probabilities is not None else None
    }
```

### Pattern 4: API Client Library Wrapper

```python
# python/api_client_wrapper.py
"""Wrap an API client library (e.g., stripe, twilio, sendgrid)"""

import stripe
from mxcp.runtime import get_secret, on_init

@on_init
def initialize_stripe():
    """Configure Stripe on startup"""
    secret = get_secret("stripe")
    if secret:
        stripe.api_key = secret["api_key"]

def create_customer(email: str, name: str) -> dict:
    """Wrap Stripe customer creation"""
    try:
        customer = stripe.Customer.create(
            email=email,
            name=name
        )
        return {
            "customer_id": customer.id,
            "email": customer.email,
            "name": customer.name,
            "created": customer.created
        }
    except stripe.error.StripeError as e:
        return {"error": str(e), "type": e.__class__.__name__}

def list_charges(customer_id: str = None, limit: int = 10) -> dict:
    """Wrap Stripe charges listing"""
    try:
        charges = stripe.Charge.list(
            customer=customer_id,
            limit=limit
        )
        return {
            "charges": [
                {
                    "id": charge.id,
                    "amount": charge.amount,
                    "currency": charge.currency,
                    "status": charge.status,
                    "created": charge.created
                }
                for charge in charges.data
            ]
        }
    except stripe.error.StripeError as e:
        return {"error": str(e)}
```

### Pattern 5: Async Library Wrapper

```python
# python/async_library_wrapper.py
"""Wrap async libraries like httpx, aiohttp"""

import httpx
import asyncio
from mxcp.runtime import get_secret

async def batch_fetch(urls: list[str]) -> list[dict]:
    """Fetch multiple URLs concurrently"""
    async with httpx.AsyncClient(timeout=30.0) as client:
        async def fetch_one(url: str) -> dict:
            try:
                response = await client.get(url)
                return {
                    "url": url,
                    "status": response.status_code,
                    "data": response.json() if response.headers.get('content-type', '').startswith('application/json') else response.text
                }
            except Exception as e:
                return {"url": url, "error": str(e)}

        results = await asyncio.gather(*[fetch_one(url) for url in urls])
        return results

async def graphql_query(endpoint: str, query: str, variables: dict = None) -> dict:
    """Wrap GraphQL library/client"""
    secret = get_secret("graphql_api")
    headers = {"Authorization": f"Bearer {secret['token']}"} if secret else {}

    async with httpx.AsyncClient() as client:
        response = await client.post(
            endpoint,
            json={"query": query, "variables": variables or {}},
            headers=headers
        )
        return response.json()
```

### Pattern 6: Complex Library with State Management

```python
# python/stateful_library_wrapper.py
"""Wrap libraries that maintain state (e.g., database connections, cache clients)"""

from redis import Redis
from mxcp.runtime import get_secret, on_init, on_shutdown

redis_client = None

@on_init
def connect_redis():
    """Initialize Redis connection on startup"""
    global redis_client
    secret = get_secret("redis")
    if secret:
        redis_client = Redis(
            host=secret["host"],
            port=secret.get("port", 6379),
            password=secret.get("password"),
            decode_responses=True
        )

@on_shutdown
def disconnect_redis():
    """Clean up Redis connection"""
    global redis_client
    if redis_client:
        redis_client.close()

def cache_set(key: str, value: str, ttl: int = 3600) -> dict:
    """Set value in Redis cache"""
    if not redis_client:
        return {"error": "Redis not configured"}

    try:
        redis_client.setex(key, ttl, value)
        return {"status": "success", "key": key, "ttl": ttl}
    except Exception as e:
        return {"error": str(e)}

def cache_get(key: str) -> dict:
    """Get value from Redis cache"""
    if not redis_client:
        return {"error": "Redis not configured"}

    try:
        value = redis_client.get(key)
        return {"key": key, "value": value, "found": value is not None}
    except Exception as e:
        return {"error": str(e)}
```

## Dependency Management

### requirements.txt

Always include dependencies for wrapped libraries:

```txt
# requirements.txt

# HTTP clients
requests>=2.31.0
httpx>=0.24.0
aiohttp>=3.8.0

# Data processing
pandas>=2.0.0
numpy>=1.24.0
openpyxl>=3.1.0  # For Excel support

# ML libraries
scikit-learn>=1.3.0

# API clients
stripe>=5.4.0
twilio>=8.0.0
sendgrid>=6.10.0

# Database/Cache
redis>=4.5.0
psycopg2-binary>=2.9.0  # For PostgreSQL

# Other common libraries
pillow>=10.0.0  # Image processing
beautifulsoup4>=4.12.0  # HTML parsing
lxml>=4.9.0  # XML parsing
```

### Installing Dependencies

```bash
# In project directory
pip install -r requirements.txt

# Or install specific library
pip install pandas requests
```

## Error Handling for Library Wrappers

**Always handle library-specific exceptions**:

```python
def safe_library_call(param: str) -> dict:
    """Template for safe library wrapping"""
    try:
        # Import library (can fail if not installed)
        import some_library

        # Use library
        result = some_library.do_something(param)

        return {"success": True, "result": result}

    except ImportError as e:
        return {
            "error": "Library not installed",
            "message": str(e),
            "fix": "Run: pip install some_library"
        }
    except some_library.SpecificError as e:
        return {
            "error": "Library-specific error",
            "message": str(e),
            "type": e.__class__.__name__
        }
    except Exception as e:
        return {
            "error": "Unexpected error",
            "message": str(e),
            "type": e.__class__.__name__
        }
```

## Database Reload (Advanced)

**Important**: In most cases, you DON'T need this feature. Use `db.execute()` directly for database operations.

The `reload_duckdb()` function allows Python endpoints to trigger a safe reload of the DuckDB database. This is **only** needed when external processes require exclusive access to the database file.

### When to Use

Use `reload_duckdb()` ONLY when:
- External tools need exclusive database access (e.g., running `dbt` as a subprocess)
- You're replacing the entire database file
- External processes cannot operate within the same Python process

### When NOT to Use

- ❌ Regular database operations (use `db.execute()` instead)
- ❌ Running dbt (use dbt Python API directly in the same process)
- ❌ Loading data from APIs/files (use `db.execute()` to insert data)

DuckDB's concurrency model allows the MXCP process to own the connection while multiple threads operate safely. Only use `reload_duckdb()` if you absolutely must have an external process update the database file.

### API

```python
from mxcp.runtime import reload_duckdb

def update_data_endpoint() -> dict:
    """Endpoint that triggers a data refresh"""

    def rebuild_database():
        """
        This function runs with all connections closed.
        You have exclusive access to the DuckDB file.
        """
        # Example: Run external tool
        import subprocess
        subprocess.run(["dbt", "run", "--target", "prod"], check=True)

        # Or: Replace with pre-built database
        import shutil
        shutil.copy("/staging/analytics.duckdb", "/app/data/analytics.duckdb")

        # Or: Load fresh data
        import pandas as pd
        import duckdb
        df = pd.read_parquet("s3://bucket/latest-data.parquet")
        conn = duckdb.connect("/app/data/analytics.duckdb")
        conn.execute("CREATE OR REPLACE TABLE sales AS SELECT * FROM df")
        conn.close()

    # Schedule the reload (happens asynchronously)
    reload_duckdb(
        payload_func=rebuild_database,
        description="Updating analytics data"
    )

    # Return immediately - reload happens in background
    return {
        "status": "scheduled",
        "message": "Data refresh will complete in background"
    }
```

### How It Works

When you call `reload_duckdb()`:

1. **Queues the reload** - Function returns immediately to client
2. **Drains active requests** - Existing requests complete normally
3. **Shuts down runtime** - Closes Python hooks and DuckDB connections
4. **Runs your payload** - With all connections closed and exclusive access
5. **Restarts runtime** - Fresh configuration and connections
6. **Processes waiting requests** - With the updated data

### Real-World Example

```python
from mxcp.runtime import reload_duckdb, db
from datetime import datetime
import requests

def scheduled_update(source: str = "api") -> dict:
    """Endpoint called by cron to update data"""

    def rebuild_from_api():
        # Fetch data from external API
        response = requests.get("https://api.example.com/analytics/export")
        data = response.json()

        # Write to DuckDB (exclusive access guaranteed)
        import duckdb
        conn = duckdb.connect("/app/data/analytics.duckdb")

        # Clear old data
        conn.execute("DROP TABLE IF EXISTS daily_metrics")

        # Load new data
        conn.execute("""
            CREATE TABLE daily_metrics AS
            SELECT * FROM read_json_auto(?)
        """, [data])

        # Update metadata
        conn.execute("""
            INSERT INTO update_log (timestamp, source, record_count)
            VALUES (?, ?, ?)
        """, [datetime.now(), source, len(data)])

        conn.close()

    reload_duckdb(
        payload_func=rebuild_from_api,
        description=f"Scheduled update from {source}"
    )

    return {
        "status": "scheduled",
        "source": source,
        "timestamp": datetime.now().isoformat()
    }
```

### Best Practices

1. **Avoid when possible** - Prefer direct `db.execute()` operations
2. **Return immediately** - Don't wait for reload in your endpoint
3. **Handle errors in payload** - Wrap payload logic in try/except
4. **Keep payload fast** - Long-running payloads block new requests
5. **Document behavior** - Let users know data refresh is asynchronous

## Plugin System

MXCP supports a plugin system for extending DuckDB with custom Python functions.

### Accessing Plugins

```python
from mxcp.runtime import plugins

# Get a specific plugin
my_plugin = plugins.get("my_custom_plugin")
if my_plugin:
    result = my_plugin.some_method()

# List available plugins
available_plugins = plugins.list()
print(f"Available plugins: {available_plugins}")
```

### Example Usage

```python
def use_custom_function(data: str) -> dict:
    """Use a custom DuckDB function from a plugin"""

    # Get the plugin
    text_plugin = plugins.get("text_processing")
    if not text_plugin:
        return {"error": "text_processing plugin not available"}

    # Use plugin functionality
    result = text_plugin.normalize_text(data)

    return {"normalized": result}
```

### Plugin Definition

Plugins are defined in `plugins/` directory:

```python
# plugins/my_plugin.py
def custom_transform(value: str) -> str:
    """Custom transformation logic"""
    return value.upper()

# Register with DuckDB if needed
def register_functions(conn):
    """Register custom functions with DuckDB"""
    conn.create_function("custom_upper", custom_transform)
```

See official MXCP documentation for complete plugin development guide.

## Best Practices for Library Wrapping

1. **Initialize once**: Use `@on_init` for expensive setup (connections, model loading)
2. **Clean up**: Use `@on_shutdown` to release resources (HTTP clients, NOT database)
3. **Handle errors**: Catch library-specific exceptions, return error dicts
4. **Document dependencies**: List in requirements.txt with versions
5. **Type hints**: Add for better IDE support and documentation
6. **Async when appropriate**: Use async for I/O-bound library operations
7. **State management**: Use global variables + lifecycle hooks for stateful clients
8. **Version pin**: Pin library versions to avoid breaking changes
9. **Timeout handling**: Add timeouts for network operations
10. **Return simple types**: Convert library-specific objects to dicts/lists

## General Best Practices

1. **Database Access**: Always use `db.execute()`, never cache connections
2. **Error Handling**: Return error dicts instead of raising exceptions
3. **Type Hints**: Use for better IDE support
4. **Logging**: Use standard Python logging
5. **Resource Management**: Use context managers
6. **Async**: Use for I/O-bound operations