335 lines
8.1 KiB
Markdown
335 lines
8.1 KiB
Markdown
# MuJoCo VR Hand Control
|
|
|
|
## Overview
|
|
|
|
This tutorial demonstrates how to control virtual hands in MuJoCo by leveraging mocap (motion capture) points that track user hand poses in VR environments.
|
|
|
|
## Important Requirement
|
|
|
|
**SSL/HTTPS Required**: VR hand tracking requires secure connections. Use ngrok or localtunnel to set up SSL.
|
|
|
|
See the [SSL Proxy WebXR tutorial](../basics/ssl-proxy-webxr.md) for setup instructions.
|
|
|
|
## Mocap Point API
|
|
|
|
The implementation uses **XR Hand Naming Conventions** to link mocap bodies with hand joints.
|
|
|
|
### Naming Format
|
|
|
|
```
|
|
"{joint}-{left | right}"
|
|
```
|
|
|
|
Examples:
|
|
- `wrist-right`
|
|
- `middle-finger-phalanx-proximal-right`
|
|
- `thumb-tip-left`
|
|
|
|
## Hand Joint Mapping
|
|
|
|
The system defines **25 distinct hand joints** (indexed 0-24):
|
|
|
|
### Joint Index Reference
|
|
|
|
```python
|
|
HAND_JOINTS = {
|
|
0: "wrist",
|
|
# Thumb (1-4)
|
|
1: "thumb-metacarpal",
|
|
2: "thumb-phalanx-proximal",
|
|
3: "thumb-phalanx-distal",
|
|
4: "thumb-tip",
|
|
# Index finger (5-9)
|
|
5: "index-finger-metacarpal",
|
|
6: "index-finger-phalanx-proximal",
|
|
7: "index-finger-phalanx-intermediate",
|
|
8: "index-finger-phalanx-distal",
|
|
9: "index-finger-tip",
|
|
# Middle finger (10-14)
|
|
10: "middle-finger-metacarpal",
|
|
11: "middle-finger-phalanx-proximal",
|
|
12: "middle-finger-phalanx-intermediate",
|
|
13: "middle-finger-phalanx-distal",
|
|
14: "middle-finger-tip",
|
|
# Ring finger (15-19)
|
|
15: "ring-finger-metacarpal",
|
|
16: "ring-finger-phalanx-proximal",
|
|
17: "ring-finger-phalanx-intermediate",
|
|
18: "ring-finger-phalanx-distal",
|
|
19: "ring-finger-tip",
|
|
# Pinky finger (20-24)
|
|
20: "pinky-finger-metacarpal",
|
|
21: "pinky-finger-phalanx-proximal",
|
|
22: "pinky-finger-phalanx-intermediate",
|
|
23: "pinky-finger-phalanx-distal",
|
|
24: "pinky-finger-tip",
|
|
}
|
|
```
|
|
|
|
## Complete Example
|
|
|
|
```python
|
|
import asyncio
|
|
from vuer import Vuer, VuerSession
|
|
from vuer.schemas import (
|
|
Scene, Fog, Sphere,
|
|
MuJoCo, ContribLoader,
|
|
Hands
|
|
)
|
|
from vuer.events import ClientEvent
|
|
|
|
app = Vuer()
|
|
|
|
# Assets for hand simulation
|
|
HAND_ASSETS = [
|
|
"/static/mujoco/hands/scene.xml",
|
|
"/static/mujoco/hands/left_hand.xml",
|
|
"/static/mujoco/hands/right_hand.xml",
|
|
"/static/mujoco/hands/palm.obj",
|
|
"/static/mujoco/hands/finger.obj",
|
|
]
|
|
|
|
@app.add_handler("ON_MUJOCO_FRAME")
|
|
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
|
|
"""Handle physics updates"""
|
|
print("ON_MUJOCO_FRAME", event.value)
|
|
|
|
# Access mocap data
|
|
mocap_pos = event.value.get("mocap_pos")
|
|
mocap_quat = event.value.get("mocap_quat")
|
|
|
|
# Process hand tracking data
|
|
if mocap_pos and mocap_quat:
|
|
# Update hand positions based on tracking
|
|
pass
|
|
|
|
@app.spawn(start=True)
|
|
async def main(session: VuerSession):
|
|
# Load MuJoCo library
|
|
session.upsert @ ContribLoader(
|
|
library="@vuer-ai/mujoco-ts",
|
|
version="0.0.24",
|
|
entry="dist/index.umd.js",
|
|
key="mujoco-loader",
|
|
)
|
|
|
|
await asyncio.sleep(2.0)
|
|
|
|
# Set up scene with hands
|
|
session.set @ Scene(
|
|
bgChildren=[
|
|
# MuJoCo default styling
|
|
Fog(color=0x2C3F57, near=10, far=20),
|
|
|
|
# Add VR hands
|
|
Hands(),
|
|
|
|
# Background sphere
|
|
Sphere(
|
|
args=[50, 10, 10],
|
|
materialType="basic",
|
|
material=dict(color=0x2C3F57, side=1),
|
|
),
|
|
],
|
|
|
|
# Initialize MuJoCo with hand model
|
|
MuJoCo(
|
|
src="/static/mujoco/hands/scene.xml",
|
|
assets=HAND_ASSETS,
|
|
scale=0.1,
|
|
key="hand-sim",
|
|
),
|
|
)
|
|
|
|
# Keep session alive
|
|
while True:
|
|
await asyncio.sleep(1.0)
|
|
|
|
app.run()
|
|
```
|
|
|
|
## Key Components
|
|
|
|
### Hands Component
|
|
|
|
```python
|
|
Hands()
|
|
```
|
|
|
|
This enables VR hand tracking, providing position and orientation data for all 25 hand joints.
|
|
|
|
### Mocap Bodies in MuJoCo XML
|
|
|
|
In your MuJoCo scene XML, define mocap bodies using the XR hand naming convention:
|
|
|
|
```xml
|
|
<mujoco>
|
|
<worldbody>
|
|
<!-- Right hand mocap bodies -->
|
|
<body name="wrist-right" mocap="true">
|
|
<geom type="sphere" size="0.02" rgba="1 0 0 0.5"/>
|
|
</body>
|
|
|
|
<body name="thumb-tip-right" mocap="true">
|
|
<geom type="sphere" size="0.01" rgba="0 1 0 0.5"/>
|
|
</body>
|
|
|
|
<body name="index-finger-tip-right" mocap="true">
|
|
<geom type="sphere" size="0.01" rgba="0 0 1 0.5"/>
|
|
</body>
|
|
|
|
<!-- Add more joints as needed -->
|
|
|
|
<!-- Left hand mocap bodies -->
|
|
<body name="wrist-left" mocap="true">
|
|
<geom type="sphere" size="0.02" rgba="1 0 0 0.5"/>
|
|
</body>
|
|
|
|
<!-- Add left hand joints -->
|
|
</worldbody>
|
|
</mujoco>
|
|
```
|
|
|
|
## Accessing Hand Data
|
|
|
|
### Method 1: Event Handler
|
|
|
|
```python
|
|
@app.add_handler("ON_MUJOCO_FRAME")
|
|
async def on_frame(event: ClientEvent, sess: VuerSession):
|
|
# Get mocap positions (3D coordinates)
|
|
mocap_pos = event.value.get("mocap_pos")
|
|
|
|
# Get mocap quaternions (orientations)
|
|
mocap_quat = event.value.get("mocap_quat")
|
|
|
|
if mocap_pos:
|
|
# mocap_pos is a list of [x, y, z] positions
|
|
# Order matches the mocap body order in XML
|
|
wrist_pos = mocap_pos[0]
|
|
thumb_tip_pos = mocap_pos[1]
|
|
# etc.
|
|
|
|
print(f"Wrist position: {wrist_pos}")
|
|
```
|
|
|
|
### Method 2: Direct Hand Tracking
|
|
|
|
```python
|
|
from vuer.events import ClientEvent
|
|
|
|
@app.add_handler("HAND_MOVE")
|
|
async def on_hand_move(event: ClientEvent, sess: VuerSession):
|
|
"""Handle hand tracking events directly"""
|
|
hand_data = event.value
|
|
|
|
# Access hand side
|
|
side = hand_data.get("side") # "left" or "right"
|
|
|
|
# Access joint positions
|
|
joints = hand_data.get("joints") # List of 25 joint positions
|
|
|
|
print(f"{side} hand moved")
|
|
print(f"Wrist: {joints[0]}")
|
|
print(f"Index tip: {joints[9]}")
|
|
```
|
|
|
|
## Creating Hand-Object Interactions
|
|
|
|
```python
|
|
# In your MuJoCo XML
|
|
<mujoco>
|
|
<worldbody>
|
|
<!-- Hand mocap bodies -->
|
|
<body name="index-finger-tip-right" mocap="true"/>
|
|
|
|
<!-- Graspable object -->
|
|
<body name="cube" pos="0 0 0.5">
|
|
<freejoint/>
|
|
<geom type="box" size="0.05 0.05 0.05" rgba="1 1 0 1"/>
|
|
</body>
|
|
|
|
<!-- Equality constraint to connect finger to object -->
|
|
<equality>
|
|
<weld body1="index-finger-tip-right" body2="cube" active="false"/>
|
|
</equality>
|
|
</worldbody>
|
|
|
|
<actuator>
|
|
<!-- Actuator to activate/deactivate weld constraint -->
|
|
<general joint="cube" dyntype="none"/>
|
|
</actuator>
|
|
</mujoco>
|
|
```
|
|
|
|
## Example: Pinch Detection
|
|
|
|
```python
|
|
import numpy as np
|
|
|
|
@app.add_handler("ON_MUJOCO_FRAME")
|
|
async def detect_pinch(event: ClientEvent, sess: VuerSession):
|
|
mocap_pos = event.value.get("mocap_pos")
|
|
|
|
if mocap_pos and len(mocap_pos) >= 10:
|
|
# Get thumb tip and index finger tip positions
|
|
thumb_tip = np.array(mocap_pos[4]) # Index 4
|
|
index_tip = np.array(mocap_pos[9]) # Index 9
|
|
|
|
# Calculate distance
|
|
distance = np.linalg.norm(thumb_tip - index_tip)
|
|
|
|
# Detect pinch
|
|
if distance < 0.02: # 2cm threshold
|
|
print("Pinch detected!")
|
|
# Trigger grasp action
|
|
```
|
|
|
|
## VR Access
|
|
|
|
1. Start server:
|
|
```bash
|
|
python your_script.py
|
|
```
|
|
|
|
2. Set up ngrok:
|
|
```bash
|
|
ngrok http 8012
|
|
```
|
|
|
|
3. Access via VR headset:
|
|
```
|
|
https://vuer.ai?ws=wss://xxxxx.ngrok.io
|
|
```
|
|
|
|
4. Enable hand tracking in your VR headset settings
|
|
|
|
## Best Practices
|
|
|
|
1. **Use XR naming convention** - Follow the exact joint naming format
|
|
2. **Define all needed mocap bodies** - Only tracked joints need mocap bodies
|
|
3. **Set appropriate scale** - Scale simulation for VR comfort (e.g., 0.1)
|
|
4. **Handle both hands** - Create separate mocap bodies for left and right
|
|
5. **Test joint mapping** - Verify each joint is tracking correctly
|
|
|
|
## Troubleshooting
|
|
|
|
### Hands not tracking
|
|
- Verify SSL is properly set up
|
|
- Check that hand tracking is enabled in VR headset
|
|
- Confirm `Hands()` component is in the scene
|
|
|
|
### Mocap bodies not moving
|
|
- Verify mocap body names match XR convention exactly
|
|
- Check that `mocap="true"` is set in XML
|
|
- Ensure body names include `-left` or `-right` suffix
|
|
|
|
### Poor tracking accuracy
|
|
- Calibrate VR headset hand tracking
|
|
- Ensure good lighting conditions
|
|
- Check for hand occlusion issues
|
|
|
|
## Source
|
|
|
|
Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_hand_control.html
|