Initial commit
This commit is contained in:
Zhongwei Li
285
docs/tutorials/physics/mocap-control.md
Normal file
285
docs/tutorials/physics/mocap-control.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# MuJoCo Motion Capture Control
|
||||
|
||||
## Overview
|
||||
This tutorial demonstrates implementing mocap (motion capture) control within a MuJoCo physics simulation using the Vuer framework for VR/mixed reality applications.
|
||||
|
||||
## Key Dependencies
|
||||
|
||||
```python
|
||||
from vuer import Vuer, VuerSession
|
||||
from vuer.schemas import (
|
||||
Scene, Fog, Sphere,
|
||||
MuJoCo, ContribLoader,
|
||||
MotionControllers, MotionControllerActuator
|
||||
)
|
||||
from vuer.events import ClientEvent
|
||||
```
|
||||
|
||||
## Important Requirement
|
||||
|
||||
**SSL/HTTPS Required**: The server requires SSL for WebXR functionality. Use ngrok or localtunnel to convert:
|
||||
- `ws://localhost:8012` → `wss://xxxxx.ngrok.io`
|
||||
- `http://localhost:8012` → `https://xxxxx.ngrok.io`
|
||||
|
||||
See the [SSL Proxy WebXR tutorial](../basics/ssl-proxy-webxr.md) for setup instructions.
|
||||
|
||||
## Complete Example: Gripper Control
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from vuer import Vuer, VuerSession
|
||||
from vuer.schemas import (
|
||||
Scene, Fog, Sphere,
|
||||
MuJoCo, ContribLoader,
|
||||
MotionControllers, MotionControllerActuator
|
||||
)
|
||||
from vuer.events import ClientEvent
|
||||
|
||||
app = Vuer()
|
||||
|
||||
# Define all assets for the simulation
|
||||
GRIPPER_ASSETS = [
|
||||
"/static/mujoco/gripper/scene.xml",
|
||||
"/static/mujoco/gripper/gripper.xml",
|
||||
"/static/mujoco/gripper/bin.xml",
|
||||
"/static/mujoco/gripper/table.xml",
|
||||
"/static/mujoco/gripper/base.obj",
|
||||
"/static/mujoco/gripper/finger.obj",
|
||||
"/static/mujoco/gripper/bin.obj",
|
||||
"/static/mujoco/gripper/table.obj",
|
||||
]
|
||||
|
||||
# Event handler for physics updates
|
||||
@app.add_handler("ON_MUJOCO_FRAME")
|
||||
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
|
||||
"""Respond to each simulation frame"""
|
||||
frame_data = event.value
|
||||
|
||||
# Access simulation state
|
||||
# qpos = frame_data.get("qpos") # Joint positions
|
||||
# qvel = frame_data.get("qvel") # Joint velocities
|
||||
# time = frame_data.get("time") # Simulation time
|
||||
|
||||
# Apply control inputs
|
||||
# Update visualization
|
||||
|
||||
@app.spawn(start=True)
|
||||
async def main(session: VuerSession):
|
||||
# Step 1: Load MuJoCo library
|
||||
session.upsert @ ContribLoader(
|
||||
library="@vuer-ai/mujoco-ts",
|
||||
version="0.0.24",
|
||||
entry="dist/index.umd.js",
|
||||
key="mujoco-loader",
|
||||
)
|
||||
|
||||
# Wait for library to load
|
||||
await asyncio.sleep(2.0)
|
||||
|
||||
# Step 2: Configure scene with VR controls
|
||||
session.set @ Scene(
|
||||
# Add fog effect (mimics MuJoCo's default styling)
|
||||
Fog(
|
||||
color=0x2C3F57,
|
||||
near=10,
|
||||
far=20,
|
||||
),
|
||||
|
||||
# Add background sphere
|
||||
Sphere(
|
||||
args=[50, 10, 10],
|
||||
materialType="basic",
|
||||
material=dict(color=0x2C3F57, side=1), # BackSide
|
||||
key="background",
|
||||
),
|
||||
|
||||
# Add VR motion controllers
|
||||
MotionControllers(),
|
||||
|
||||
# Add motion controller actuator for VR input
|
||||
MotionControllerActuator(
|
||||
key="controller-actuator",
|
||||
),
|
||||
|
||||
# Initialize MuJoCo simulation
|
||||
MuJoCo(
|
||||
src="/static/mujoco/gripper/scene.xml",
|
||||
assets=GRIPPER_ASSETS,
|
||||
scale=0.1,
|
||||
timeout=100,
|
||||
key="gripper-sim",
|
||||
),
|
||||
)
|
||||
|
||||
# Keep session alive
|
||||
while True:
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
app.run()
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### MotionControllers
|
||||
Captures VR controller input:
|
||||
|
||||
```python
|
||||
MotionControllers()
|
||||
```
|
||||
|
||||
This enables tracking of VR controller positions, orientations, and button presses.
|
||||
|
||||
### MotionControllerActuator
|
||||
Bridges VR input to MuJoCo simulation:
|
||||
|
||||
```python
|
||||
MotionControllerActuator(
|
||||
key="controller-actuator",
|
||||
)
|
||||
```
|
||||
|
||||
### MuJoCo Component with Scale
|
||||
```python
|
||||
MuJoCo(
|
||||
src="/static/scene.xml",
|
||||
assets=ASSETS,
|
||||
scale=0.1, # Scale simulation (10% of original size)
|
||||
timeout=100, # Timeout in milliseconds
|
||||
key="sim",
|
||||
)
|
||||
```
|
||||
|
||||
## Event Handling: ON_MUJOCO_FRAME
|
||||
|
||||
This event fires on every physics update:
|
||||
|
||||
```python
|
||||
@app.add_handler("ON_MUJOCO_FRAME")
|
||||
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
|
||||
frame_data = event.value
|
||||
|
||||
# Simulation state
|
||||
qpos = frame_data.get("qpos") # Joint positions
|
||||
qvel = frame_data.get("qvel") # Joint velocities
|
||||
time = frame_data.get("time") # Simulation time
|
||||
ctrl = frame_data.get("ctrl") # Control inputs
|
||||
|
||||
# Apply control logic
|
||||
new_ctrl = calculate_control(qpos, qvel)
|
||||
|
||||
# Update simulation
|
||||
sess.upsert @ MuJoCo(
|
||||
ctrl=new_ctrl,
|
||||
key="gripper-sim",
|
||||
)
|
||||
```
|
||||
|
||||
## Scene Setup Pattern
|
||||
|
||||
### 1. Configure MuJoCo Styling
|
||||
|
||||
```python
|
||||
Fog(
|
||||
color=0x2C3F57, # MuJoCo default gray-blue
|
||||
near=10,
|
||||
far=20,
|
||||
)
|
||||
|
||||
Sphere(
|
||||
args=[50, 10, 10],
|
||||
materialType="basic",
|
||||
material=dict(color=0x2C3F57, side=1),
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Add VR Input
|
||||
|
||||
```python
|
||||
MotionControllers()
|
||||
MotionControllerActuator()
|
||||
```
|
||||
|
||||
### 3. Initialize Physics
|
||||
|
||||
```python
|
||||
MuJoCo(
|
||||
src="/static/scene.xml",
|
||||
assets=ASSETS,
|
||||
scale=0.1,
|
||||
)
|
||||
```
|
||||
|
||||
## Asset Organization
|
||||
|
||||
Organize your gripper assets:
|
||||
|
||||
```
|
||||
static/mujoco/gripper/
|
||||
├── scene.xml # Main scene
|
||||
├── gripper.xml # Gripper model
|
||||
├── bin.xml # Bin configuration
|
||||
├── table.xml # Table configuration
|
||||
├── base.obj # 3D meshes
|
||||
├── finger.obj
|
||||
├── bin.obj
|
||||
└── table.obj
|
||||
```
|
||||
|
||||
## VR Access
|
||||
|
||||
1. Start the server:
|
||||
```bash
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
2. Set up ngrok:
|
||||
```bash
|
||||
ngrok http 8012
|
||||
```
|
||||
|
||||
3. Access via VR headset:
|
||||
```
|
||||
https://vuer.ai?ws=wss://xxxxx.ngrok.io
|
||||
```
|
||||
|
||||
## Controlling the Simulation
|
||||
|
||||
### Method 1: Direct Control Values
|
||||
|
||||
```python
|
||||
session.upsert @ MuJoCo(
|
||||
ctrl=[0.5, -0.3, 0.0], # Control values for actuators
|
||||
key="gripper-sim",
|
||||
)
|
||||
```
|
||||
|
||||
### Method 2: VR Controller Input
|
||||
|
||||
The `MotionControllerActuator` automatically maps VR controller movements to simulation controls.
|
||||
|
||||
### Method 3: Event-Based Control
|
||||
|
||||
```python
|
||||
@app.add_handler("ON_MUJOCO_FRAME")
|
||||
async def on_frame(event, sess):
|
||||
# Read current state
|
||||
qpos = event.value.get("qpos")
|
||||
|
||||
# Calculate control
|
||||
ctrl = your_control_algorithm(qpos)
|
||||
|
||||
# Apply control
|
||||
sess.upsert @ MuJoCo(ctrl=ctrl, key="gripper-sim")
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use SSL** - Required for WebXR functionality
|
||||
2. **Add delays** - Wait for library to load before initializing simulation
|
||||
3. **Handle events** - Use ON_MUJOCO_FRAME for responsive control
|
||||
4. **Scale appropriately** - Adjust simulation scale for VR comfort
|
||||
5. **Declare all assets** - Include every file in the assets list
|
||||
|
||||
## Source
|
||||
|
||||
Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_control.html
|
||||
Reference in New Issue
Block a user