286 lines
6.4 KiB
Markdown
286 lines
6.4 KiB
Markdown
# MuJoCo Motion Capture Control
|
|
|
|
## Overview
|
|
This tutorial demonstrates implementing mocap (motion capture) control within a MuJoCo physics simulation using the Vuer framework for VR/mixed reality applications.
|
|
|
|
## Key Dependencies
|
|
|
|
```python
|
|
from vuer import Vuer, VuerSession
|
|
from vuer.schemas import (
|
|
Scene, Fog, Sphere,
|
|
MuJoCo, ContribLoader,
|
|
MotionControllers, MotionControllerActuator
|
|
)
|
|
from vuer.events import ClientEvent
|
|
```
|
|
|
|
## Important Requirement
|
|
|
|
**SSL/HTTPS Required**: The server requires SSL for WebXR functionality. Use ngrok or localtunnel to convert:
|
|
- `ws://localhost:8012` → `wss://xxxxx.ngrok.io`
|
|
- `http://localhost:8012` → `https://xxxxx.ngrok.io`
|
|
|
|
See the [SSL Proxy WebXR tutorial](../basics/ssl-proxy-webxr.md) for setup instructions.
|
|
|
|
## Complete Example: Gripper Control
|
|
|
|
```python
|
|
import asyncio
|
|
from vuer import Vuer, VuerSession
|
|
from vuer.schemas import (
|
|
Scene, Fog, Sphere,
|
|
MuJoCo, ContribLoader,
|
|
MotionControllers, MotionControllerActuator
|
|
)
|
|
from vuer.events import ClientEvent
|
|
|
|
app = Vuer()
|
|
|
|
# Define all assets for the simulation
|
|
GRIPPER_ASSETS = [
|
|
"/static/mujoco/gripper/scene.xml",
|
|
"/static/mujoco/gripper/gripper.xml",
|
|
"/static/mujoco/gripper/bin.xml",
|
|
"/static/mujoco/gripper/table.xml",
|
|
"/static/mujoco/gripper/base.obj",
|
|
"/static/mujoco/gripper/finger.obj",
|
|
"/static/mujoco/gripper/bin.obj",
|
|
"/static/mujoco/gripper/table.obj",
|
|
]
|
|
|
|
# Event handler for physics updates
|
|
@app.add_handler("ON_MUJOCO_FRAME")
|
|
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
|
|
"""Respond to each simulation frame"""
|
|
frame_data = event.value
|
|
|
|
# Access simulation state
|
|
# qpos = frame_data.get("qpos") # Joint positions
|
|
# qvel = frame_data.get("qvel") # Joint velocities
|
|
# time = frame_data.get("time") # Simulation time
|
|
|
|
# Apply control inputs
|
|
# Update visualization
|
|
|
|
@app.spawn(start=True)
|
|
async def main(session: VuerSession):
|
|
# Step 1: Load MuJoCo library
|
|
session.upsert @ ContribLoader(
|
|
library="@vuer-ai/mujoco-ts",
|
|
version="0.0.24",
|
|
entry="dist/index.umd.js",
|
|
key="mujoco-loader",
|
|
)
|
|
|
|
# Wait for library to load
|
|
await asyncio.sleep(2.0)
|
|
|
|
# Step 2: Configure scene with VR controls
|
|
session.set @ Scene(
|
|
# Add fog effect (mimics MuJoCo's default styling)
|
|
Fog(
|
|
color=0x2C3F57,
|
|
near=10,
|
|
far=20,
|
|
),
|
|
|
|
# Add background sphere
|
|
Sphere(
|
|
args=[50, 10, 10],
|
|
materialType="basic",
|
|
material=dict(color=0x2C3F57, side=1), # BackSide
|
|
key="background",
|
|
),
|
|
|
|
# Add VR motion controllers
|
|
MotionControllers(),
|
|
|
|
# Add motion controller actuator for VR input
|
|
MotionControllerActuator(
|
|
key="controller-actuator",
|
|
),
|
|
|
|
# Initialize MuJoCo simulation
|
|
MuJoCo(
|
|
src="/static/mujoco/gripper/scene.xml",
|
|
assets=GRIPPER_ASSETS,
|
|
scale=0.1,
|
|
timeout=100,
|
|
key="gripper-sim",
|
|
),
|
|
)
|
|
|
|
# Keep session alive
|
|
while True:
|
|
await asyncio.sleep(1.0)
|
|
|
|
app.run()
|
|
```
|
|
|
|
## Key Components
|
|
|
|
### MotionControllers
|
|
Captures VR controller input:
|
|
|
|
```python
|
|
MotionControllers()
|
|
```
|
|
|
|
This enables tracking of VR controller positions, orientations, and button presses.
|
|
|
|
### MotionControllerActuator
|
|
Bridges VR input to MuJoCo simulation:
|
|
|
|
```python
|
|
MotionControllerActuator(
|
|
key="controller-actuator",
|
|
)
|
|
```
|
|
|
|
### MuJoCo Component with Scale
|
|
```python
|
|
MuJoCo(
|
|
src="/static/scene.xml",
|
|
assets=ASSETS,
|
|
scale=0.1, # Scale simulation (10% of original size)
|
|
timeout=100, # Timeout in milliseconds
|
|
key="sim",
|
|
)
|
|
```
|
|
|
|
## Event Handling: ON_MUJOCO_FRAME
|
|
|
|
This event fires on every physics update:
|
|
|
|
```python
|
|
@app.add_handler("ON_MUJOCO_FRAME")
|
|
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
|
|
frame_data = event.value
|
|
|
|
# Simulation state
|
|
qpos = frame_data.get("qpos") # Joint positions
|
|
qvel = frame_data.get("qvel") # Joint velocities
|
|
time = frame_data.get("time") # Simulation time
|
|
ctrl = frame_data.get("ctrl") # Control inputs
|
|
|
|
# Apply control logic
|
|
new_ctrl = calculate_control(qpos, qvel)
|
|
|
|
# Update simulation
|
|
sess.upsert @ MuJoCo(
|
|
ctrl=new_ctrl,
|
|
key="gripper-sim",
|
|
)
|
|
```
|
|
|
|
## Scene Setup Pattern
|
|
|
|
### 1. Configure MuJoCo Styling
|
|
|
|
```python
|
|
Fog(
|
|
color=0x2C3F57, # MuJoCo default gray-blue
|
|
near=10,
|
|
far=20,
|
|
)
|
|
|
|
Sphere(
|
|
args=[50, 10, 10],
|
|
materialType="basic",
|
|
material=dict(color=0x2C3F57, side=1),
|
|
)
|
|
```
|
|
|
|
### 2. Add VR Input
|
|
|
|
```python
|
|
MotionControllers()
|
|
MotionControllerActuator()
|
|
```
|
|
|
|
### 3. Initialize Physics
|
|
|
|
```python
|
|
MuJoCo(
|
|
src="/static/scene.xml",
|
|
assets=ASSETS,
|
|
scale=0.1,
|
|
)
|
|
```
|
|
|
|
## Asset Organization
|
|
|
|
Organize your gripper assets:
|
|
|
|
```
|
|
static/mujoco/gripper/
|
|
├── scene.xml # Main scene
|
|
├── gripper.xml # Gripper model
|
|
├── bin.xml # Bin configuration
|
|
├── table.xml # Table configuration
|
|
├── base.obj # 3D meshes
|
|
├── finger.obj
|
|
├── bin.obj
|
|
└── table.obj
|
|
```
|
|
|
|
## VR Access
|
|
|
|
1. Start the server:
|
|
```bash
|
|
python your_script.py
|
|
```
|
|
|
|
2. Set up ngrok:
|
|
```bash
|
|
ngrok http 8012
|
|
```
|
|
|
|
3. Access via VR headset:
|
|
```
|
|
https://vuer.ai?ws=wss://xxxxx.ngrok.io
|
|
```
|
|
|
|
## Controlling the Simulation
|
|
|
|
### Method 1: Direct Control Values
|
|
|
|
```python
|
|
session.upsert @ MuJoCo(
|
|
ctrl=[0.5, -0.3, 0.0], # Control values for actuators
|
|
key="gripper-sim",
|
|
)
|
|
```
|
|
|
|
### Method 2: VR Controller Input
|
|
|
|
The `MotionControllerActuator` automatically maps VR controller movements to simulation controls.
|
|
|
|
### Method 3: Event-Based Control
|
|
|
|
```python
|
|
@app.add_handler("ON_MUJOCO_FRAME")
|
|
async def on_frame(event, sess):
|
|
# Read current state
|
|
qpos = event.value.get("qpos")
|
|
|
|
# Calculate control
|
|
ctrl = your_control_algorithm(qpos)
|
|
|
|
# Apply control
|
|
sess.upsert @ MuJoCo(ctrl=ctrl, key="gripper-sim")
|
|
```
|
|
|
|
## Best Practices
|
|
|
|
1. **Use SSL** - Required for WebXR functionality
|
|
2. **Add delays** - Wait for library to load before initializing simulation
|
|
3. **Handle events** - Use ON_MUJOCO_FRAME for responsive control
|
|
4. **Scale appropriately** - Adjust simulation scale for VR comfort
|
|
5. **Declare all assets** - Include every file in the assets list
|
|
|
|
## Source
|
|
|
|
Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_control.html
|