# MuJoCo Motion Capture Control

## Overview
This tutorial demonstrates implementing mocap (motion capture) control within a MuJoCo physics simulation using the Vuer framework for VR/mixed reality applications.

## Key Dependencies

```python
from vuer import Vuer, VuerSession
from vuer.schemas import (
    Scene, Fog, Sphere,
    MuJoCo, ContribLoader,
    MotionControllers, MotionControllerActuator
)
from vuer.events import ClientEvent
```

## Important Requirement

**SSL/HTTPS Required**: The server requires SSL for WebXR functionality. Use ngrok or localtunnel to convert:
- `ws://localhost:8012` → `wss://xxxxx.ngrok.io`
- `http://localhost:8012` → `https://xxxxx.ngrok.io`

See the [SSL Proxy WebXR tutorial](../basics/ssl-proxy-webxr.md) for setup instructions.

## Complete Example: Gripper Control

```python
import asyncio
from vuer import Vuer, VuerSession
from vuer.schemas import (
    Scene, Fog, Sphere,
    MuJoCo, ContribLoader,
    MotionControllers, MotionControllerActuator
)
from vuer.events import ClientEvent

app = Vuer()

# Define all assets for the simulation
GRIPPER_ASSETS = [
    "/static/mujoco/gripper/scene.xml",
    "/static/mujoco/gripper/gripper.xml",
    "/static/mujoco/gripper/bin.xml",
    "/static/mujoco/gripper/table.xml",
    "/static/mujoco/gripper/base.obj",
    "/static/mujoco/gripper/finger.obj",
    "/static/mujoco/gripper/bin.obj",
    "/static/mujoco/gripper/table.obj",
]

# Event handler for physics updates
@app.add_handler("ON_MUJOCO_FRAME")
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
    """Respond to each simulation frame"""
    frame_data = event.value

    # Access simulation state
    # qpos = frame_data.get("qpos")  # Joint positions
    # qvel = frame_data.get("qvel")  # Joint velocities
    # time = frame_data.get("time")  # Simulation time

    # Apply control inputs
    # Update visualization

@app.spawn(start=True)
async def main(session: VuerSession):
    # Step 1: Load MuJoCo library
    session.upsert @ ContribLoader(
        library="@vuer-ai/mujoco-ts",
        version="0.0.24",
        entry="dist/index.umd.js",
        key="mujoco-loader",
    )

    # Wait for library to load
    await asyncio.sleep(2.0)

    # Step 2: Configure scene with VR controls
    session.set @ Scene(
        # Add fog effect (mimics MuJoCo's default styling)
        Fog(
            color=0x2C3F57,
            near=10,
            far=20,
        ),

        # Add background sphere
        Sphere(
            args=[50, 10, 10],
            materialType="basic",
            material=dict(color=0x2C3F57, side=1),  # BackSide
            key="background",
        ),

        # Add VR motion controllers
        MotionControllers(),

        # Add motion controller actuator for VR input
        MotionControllerActuator(
            key="controller-actuator",
        ),

        # Initialize MuJoCo simulation
        MuJoCo(
            src="/static/mujoco/gripper/scene.xml",
            assets=GRIPPER_ASSETS,
            scale=0.1,
            timeout=100,
            key="gripper-sim",
        ),
    )

    # Keep session alive
    while True:
        await asyncio.sleep(1.0)

app.run()
```

## Key Components

### MotionControllers
Captures VR controller input:

```python
MotionControllers()
```

This enables tracking of VR controller positions, orientations, and button presses.

### MotionControllerActuator
Bridges VR input to MuJoCo simulation:

```python
MotionControllerActuator(
    key="controller-actuator",
)
```

### MuJoCo Component with Scale
```python
MuJoCo(
    src="/static/scene.xml",
    assets=ASSETS,
    scale=0.1,      # Scale simulation (10% of original size)
    timeout=100,    # Timeout in milliseconds
    key="sim",
)
```

## Event Handling: ON_MUJOCO_FRAME

This event fires on every physics update:

```python
@app.add_handler("ON_MUJOCO_FRAME")
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
    frame_data = event.value

    # Simulation state
    qpos = frame_data.get("qpos")      # Joint positions
    qvel = frame_data.get("qvel")      # Joint velocities
    time = frame_data.get("time")      # Simulation time
    ctrl = frame_data.get("ctrl")      # Control inputs

    # Apply control logic
    new_ctrl = calculate_control(qpos, qvel)

    # Update simulation
    sess.upsert @ MuJoCo(
        ctrl=new_ctrl,
        key="gripper-sim",
    )
```

## Scene Setup Pattern

### 1. Configure MuJoCo Styling

```python
Fog(
    color=0x2C3F57,  # MuJoCo default gray-blue
    near=10,
    far=20,
)

Sphere(
    args=[50, 10, 10],
    materialType="basic",
    material=dict(color=0x2C3F57, side=1),
)
```

### 2. Add VR Input

```python
MotionControllers()
MotionControllerActuator()
```

### 3. Initialize Physics

```python
MuJoCo(
    src="/static/scene.xml",
    assets=ASSETS,
    scale=0.1,
)
```

## Asset Organization

Organize your gripper assets:

```
static/mujoco/gripper/
├── scene.xml          # Main scene
├── gripper.xml        # Gripper model
├── bin.xml            # Bin configuration
├── table.xml          # Table configuration
├── base.obj           # 3D meshes
├── finger.obj
├── bin.obj
└── table.obj
```

## VR Access

1. Start the server:
```bash
python your_script.py
```

2. Set up ngrok:
```bash
ngrok http 8012
```

3. Access via VR headset:
```
https://vuer.ai?ws=wss://xxxxx.ngrok.io
```

## Controlling the Simulation

### Method 1: Direct Control Values

```python
session.upsert @ MuJoCo(
    ctrl=[0.5, -0.3, 0.0],  # Control values for actuators
    key="gripper-sim",
)
```

### Method 2: VR Controller Input

The `MotionControllerActuator` automatically maps VR controller movements to simulation controls.

### Method 3: Event-Based Control

```python
@app.add_handler("ON_MUJOCO_FRAME")
async def on_frame(event, sess):
    # Read current state
    qpos = event.value.get("qpos")

    # Calculate control
    ctrl = your_control_algorithm(qpos)

    # Apply control
    sess.upsert @ MuJoCo(ctrl=ctrl, key="gripper-sim")
```

## Best Practices

1. **Use SSL** - Required for WebXR functionality
2. **Add delays** - Wait for library to load before initializing simulation
3. **Handle events** - Use ON_MUJOCO_FRAME for responsive control
4. **Scale appropriately** - Adjust simulation scale for VR comfort
5. **Declare all assets** - Include every file in the assets list

## Source

Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_control.html