gh-vuer-ai-vuer-skill-marketplace-skills-docs-vuer-ai/docs/tutorials/physics/mocap-control.md

MuJoCo Motion Capture Control

Overview

This tutorial demonstrates implementing mocap (motion capture) control within a MuJoCo physics simulation using the Vuer framework for VR/mixed reality applications.

Key Dependencies

from vuer import Vuer, VuerSession
from vuer.schemas import (
    Scene, Fog, Sphere,
    MuJoCo, ContribLoader,
    MotionControllers, MotionControllerActuator
)
from vuer.events import ClientEvent

Important Requirement

SSL/HTTPS Required: The server requires SSL for WebXR functionality. Use ngrok or localtunnel to convert:

• ws://localhost:8012 → wss://xxxxx.ngrok.io
• http://localhost:8012 → https://xxxxx.ngrok.io

See the SSL Proxy WebXR tutorial for setup instructions.

Complete Example: Gripper Control

import asyncio
from vuer import Vuer, VuerSession
from vuer.schemas import (
    Scene, Fog, Sphere,
    MuJoCo, ContribLoader,
    MotionControllers, MotionControllerActuator
)
from vuer.events import ClientEvent

app = Vuer()

# Define all assets for the simulation
GRIPPER_ASSETS = [
    "/static/mujoco/gripper/scene.xml",
    "/static/mujoco/gripper/gripper.xml",
    "/static/mujoco/gripper/bin.xml",
    "/static/mujoco/gripper/table.xml",
    "/static/mujoco/gripper/base.obj",
    "/static/mujoco/gripper/finger.obj",
    "/static/mujoco/gripper/bin.obj",
    "/static/mujoco/gripper/table.obj",
]

# Event handler for physics updates
@app.add_handler("ON_MUJOCO_FRAME")
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
    """Respond to each simulation frame"""
    frame_data = event.value

    # Access simulation state
    # qpos = frame_data.get("qpos")  # Joint positions
    # qvel = frame_data.get("qvel")  # Joint velocities
    # time = frame_data.get("time")  # Simulation time

    # Apply control inputs
    # Update visualization

@app.spawn(start=True)
async def main(session: VuerSession):
    # Step 1: Load MuJoCo library
    session.upsert @ ContribLoader(
        library="@vuer-ai/mujoco-ts",
        version="0.0.24",
        entry="dist/index.umd.js",
        key="mujoco-loader",
    )

    # Wait for library to load
    await asyncio.sleep(2.0)

    # Step 2: Configure scene with VR controls
    session.set @ Scene(
        # Add fog effect (mimics MuJoCo's default styling)
        Fog(
            color=0x2C3F57,
            near=10,
            far=20,
        ),

        # Add background sphere
        Sphere(
            args=[50, 10, 10],
            materialType="basic",
            material=dict(color=0x2C3F57, side=1),  # BackSide
            key="background",
        ),

        # Add VR motion controllers
        MotionControllers(),

        # Add motion controller actuator for VR input
        MotionControllerActuator(
            key="controller-actuator",
        ),

        # Initialize MuJoCo simulation
        MuJoCo(
            src="/static/mujoco/gripper/scene.xml",
            assets=GRIPPER_ASSETS,
            scale=0.1,
            timeout=100,
            key="gripper-sim",
        ),
    )

    # Keep session alive
    while True:
        await asyncio.sleep(1.0)

app.run()

Key Components

MotionControllers

Captures VR controller input:

MotionControllers()

This enables tracking of VR controller positions, orientations, and button presses.

MotionControllerActuator

Bridges VR input to MuJoCo simulation:

MotionControllerActuator(
    key="controller-actuator",
)

MuJoCo Component with Scale

MuJoCo(
    src="/static/scene.xml",
    assets=ASSETS,
    scale=0.1,      # Scale simulation (10% of original size)
    timeout=100,    # Timeout in milliseconds
    key="sim",
)

Event Handling: ON_MUJOCO_FRAME

This event fires on every physics update:

@app.add_handler("ON_MUJOCO_FRAME")
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
    frame_data = event.value

    # Simulation state
    qpos = frame_data.get("qpos")      # Joint positions
    qvel = frame_data.get("qvel")      # Joint velocities
    time = frame_data.get("time")      # Simulation time
    ctrl = frame_data.get("ctrl")      # Control inputs

    # Apply control logic
    new_ctrl = calculate_control(qpos, qvel)

    # Update simulation
    sess.upsert @ MuJoCo(
        ctrl=new_ctrl,
        key="gripper-sim",
    )

Scene Setup Pattern

1. Configure MuJoCo Styling

Fog(
    color=0x2C3F57,  # MuJoCo default gray-blue
    near=10,
    far=20,
)

Sphere(
    args=[50, 10, 10],
    materialType="basic",
    material=dict(color=0x2C3F57, side=1),
)

2. Add VR Input

MotionControllers()
MotionControllerActuator()

3. Initialize Physics

MuJoCo(
    src="/static/scene.xml",
    assets=ASSETS,
    scale=0.1,
)

Asset Organization

Organize your gripper assets:

static/mujoco/gripper/
├── scene.xml          # Main scene
├── gripper.xml        # Gripper model
├── bin.xml            # Bin configuration
├── table.xml          # Table configuration
├── base.obj           # 3D meshes
├── finger.obj
├── bin.obj
└── table.obj

VR Access

1. Start the server:

python your_script.py

2. Set up ngrok:

ngrok http 8012

3. Access via VR headset:

https://vuer.ai?ws=wss://xxxxx.ngrok.io

Controlling the Simulation

Method 1: Direct Control Values

session.upsert @ MuJoCo(
    ctrl=[0.5, -0.3, 0.0],  # Control values for actuators
    key="gripper-sim",
)

Method 2: VR Controller Input

The MotionControllerActuator automatically maps VR controller movements to simulation controls.

Method 3: Event-Based Control

@app.add_handler("ON_MUJOCO_FRAME")
async def on_frame(event, sess):
    # Read current state
    qpos = event.value.get("qpos")

    # Calculate control
    ctrl = your_control_algorithm(qpos)

    # Apply control
    sess.upsert @ MuJoCo(ctrl=ctrl, key="gripper-sim")

Best Practices

1. Use SSL - Required for WebXR functionality
2. Add delays - Wait for library to load before initializing simulation
3. Handle events - Use ON_MUJOCO_FRAME for responsive control
4. Scale appropriately - Adjust simulation scale for VR comfort
5. Declare all assets - Include every file in the assets list

Source

Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_control.html