Initial commit
This commit is contained in:
Zhongwei Li
334
docs/tutorials/physics/hand-control.md
Normal file
334
docs/tutorials/physics/hand-control.md
Normal file
@@ -0,0 +1,334 @@
|
||||
# MuJoCo VR Hand Control
|
||||
|
||||
## Overview
|
||||
|
||||
This tutorial demonstrates how to control virtual hands in MuJoCo by leveraging mocap (motion capture) points that track user hand poses in VR environments.
|
||||
|
||||
## Important Requirement
|
||||
|
||||
**SSL/HTTPS Required**: VR hand tracking requires secure connections. Use ngrok or localtunnel to set up SSL.
|
||||
|
||||
See the [SSL Proxy WebXR tutorial](../basics/ssl-proxy-webxr.md) for setup instructions.
|
||||
|
||||
## Mocap Point API
|
||||
|
||||
The implementation uses **XR Hand Naming Conventions** to link mocap bodies with hand joints.
|
||||
|
||||
### Naming Format
|
||||
|
||||
```
|
||||
"{joint}-{left | right}"
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `wrist-right`
|
||||
- `middle-finger-phalanx-proximal-right`
|
||||
- `thumb-tip-left`
|
||||
|
||||
## Hand Joint Mapping
|
||||
|
||||
The system defines **25 distinct hand joints** (indexed 0-24):
|
||||
|
||||
### Joint Index Reference
|
||||
|
||||
```python
|
||||
HAND_JOINTS = {
|
||||
0: "wrist",
|
||||
# Thumb (1-4)
|
||||
1: "thumb-metacarpal",
|
||||
2: "thumb-phalanx-proximal",
|
||||
3: "thumb-phalanx-distal",
|
||||
4: "thumb-tip",
|
||||
# Index finger (5-9)
|
||||
5: "index-finger-metacarpal",
|
||||
6: "index-finger-phalanx-proximal",
|
||||
7: "index-finger-phalanx-intermediate",
|
||||
8: "index-finger-phalanx-distal",
|
||||
9: "index-finger-tip",
|
||||
# Middle finger (10-14)
|
||||
10: "middle-finger-metacarpal",
|
||||
11: "middle-finger-phalanx-proximal",
|
||||
12: "middle-finger-phalanx-intermediate",
|
||||
13: "middle-finger-phalanx-distal",
|
||||
14: "middle-finger-tip",
|
||||
# Ring finger (15-19)
|
||||
15: "ring-finger-metacarpal",
|
||||
16: "ring-finger-phalanx-proximal",
|
||||
17: "ring-finger-phalanx-intermediate",
|
||||
18: "ring-finger-phalanx-distal",
|
||||
19: "ring-finger-tip",
|
||||
# Pinky finger (20-24)
|
||||
20: "pinky-finger-metacarpal",
|
||||
21: "pinky-finger-phalanx-proximal",
|
||||
22: "pinky-finger-phalanx-intermediate",
|
||||
23: "pinky-finger-phalanx-distal",
|
||||
24: "pinky-finger-tip",
|
||||
}
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from vuer import Vuer, VuerSession
|
||||
from vuer.schemas import (
|
||||
Scene, Fog, Sphere,
|
||||
MuJoCo, ContribLoader,
|
||||
Hands
|
||||
)
|
||||
from vuer.events import ClientEvent
|
||||
|
||||
app = Vuer()
|
||||
|
||||
# Assets for hand simulation
|
||||
HAND_ASSETS = [
|
||||
"/static/mujoco/hands/scene.xml",
|
||||
"/static/mujoco/hands/left_hand.xml",
|
||||
"/static/mujoco/hands/right_hand.xml",
|
||||
"/static/mujoco/hands/palm.obj",
|
||||
"/static/mujoco/hands/finger.obj",
|
||||
]
|
||||
|
||||
@app.add_handler("ON_MUJOCO_FRAME")
|
||||
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
|
||||
"""Handle physics updates"""
|
||||
print("ON_MUJOCO_FRAME", event.value)
|
||||
|
||||
# Access mocap data
|
||||
mocap_pos = event.value.get("mocap_pos")
|
||||
mocap_quat = event.value.get("mocap_quat")
|
||||
|
||||
# Process hand tracking data
|
||||
if mocap_pos and mocap_quat:
|
||||
# Update hand positions based on tracking
|
||||
pass
|
||||
|
||||
@app.spawn(start=True)
|
||||
async def main(session: VuerSession):
|
||||
# Load MuJoCo library
|
||||
session.upsert @ ContribLoader(
|
||||
library="@vuer-ai/mujoco-ts",
|
||||
version="0.0.24",
|
||||
entry="dist/index.umd.js",
|
||||
key="mujoco-loader",
|
||||
)
|
||||
|
||||
await asyncio.sleep(2.0)
|
||||
|
||||
# Set up scene with hands
|
||||
session.set @ Scene(
|
||||
bgChildren=[
|
||||
# MuJoCo default styling
|
||||
Fog(color=0x2C3F57, near=10, far=20),
|
||||
|
||||
# Add VR hands
|
||||
Hands(),
|
||||
|
||||
# Background sphere
|
||||
Sphere(
|
||||
args=[50, 10, 10],
|
||||
materialType="basic",
|
||||
material=dict(color=0x2C3F57, side=1),
|
||||
),
|
||||
],
|
||||
|
||||
# Initialize MuJoCo with hand model
|
||||
MuJoCo(
|
||||
src="/static/mujoco/hands/scene.xml",
|
||||
assets=HAND_ASSETS,
|
||||
scale=0.1,
|
||||
key="hand-sim",
|
||||
),
|
||||
)
|
||||
|
||||
# Keep session alive
|
||||
while True:
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
app.run()
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### Hands Component
|
||||
|
||||
```python
|
||||
Hands()
|
||||
```
|
||||
|
||||
This enables VR hand tracking, providing position and orientation data for all 25 hand joints.
|
||||
|
||||
### Mocap Bodies in MuJoCo XML
|
||||
|
||||
In your MuJoCo scene XML, define mocap bodies using the XR hand naming convention:
|
||||
|
||||
```xml
|
||||
<mujoco>
|
||||
<worldbody>
|
||||
<!-- Right hand mocap bodies -->
|
||||
<body name="wrist-right" mocap="true">
|
||||
<geom type="sphere" size="0.02" rgba="1 0 0 0.5"/>
|
||||
</body>
|
||||
|
||||
<body name="thumb-tip-right" mocap="true">
|
||||
<geom type="sphere" size="0.01" rgba="0 1 0 0.5"/>
|
||||
</body>
|
||||
|
||||
<body name="index-finger-tip-right" mocap="true">
|
||||
<geom type="sphere" size="0.01" rgba="0 0 1 0.5"/>
|
||||
</body>
|
||||
|
||||
<!-- Add more joints as needed -->
|
||||
|
||||
<!-- Left hand mocap bodies -->
|
||||
<body name="wrist-left" mocap="true">
|
||||
<geom type="sphere" size="0.02" rgba="1 0 0 0.5"/>
|
||||
</body>
|
||||
|
||||
<!-- Add left hand joints -->
|
||||
</worldbody>
|
||||
</mujoco>
|
||||
```
|
||||
|
||||
## Accessing Hand Data
|
||||
|
||||
### Method 1: Event Handler
|
||||
|
||||
```python
|
||||
@app.add_handler("ON_MUJOCO_FRAME")
|
||||
async def on_frame(event: ClientEvent, sess: VuerSession):
|
||||
# Get mocap positions (3D coordinates)
|
||||
mocap_pos = event.value.get("mocap_pos")
|
||||
|
||||
# Get mocap quaternions (orientations)
|
||||
mocap_quat = event.value.get("mocap_quat")
|
||||
|
||||
if mocap_pos:
|
||||
# mocap_pos is a list of [x, y, z] positions
|
||||
# Order matches the mocap body order in XML
|
||||
wrist_pos = mocap_pos[0]
|
||||
thumb_tip_pos = mocap_pos[1]
|
||||
# etc.
|
||||
|
||||
print(f"Wrist position: {wrist_pos}")
|
||||
```
|
||||
|
||||
### Method 2: Direct Hand Tracking
|
||||
|
||||
```python
|
||||
from vuer.events import ClientEvent
|
||||
|
||||
@app.add_handler("HAND_MOVE")
|
||||
async def on_hand_move(event: ClientEvent, sess: VuerSession):
|
||||
"""Handle hand tracking events directly"""
|
||||
hand_data = event.value
|
||||
|
||||
# Access hand side
|
||||
side = hand_data.get("side") # "left" or "right"
|
||||
|
||||
# Access joint positions
|
||||
joints = hand_data.get("joints") # List of 25 joint positions
|
||||
|
||||
print(f"{side} hand moved")
|
||||
print(f"Wrist: {joints[0]}")
|
||||
print(f"Index tip: {joints[9]}")
|
||||
```
|
||||
|
||||
## Creating Hand-Object Interactions
|
||||
|
||||
```python
|
||||
# In your MuJoCo XML
|
||||
<mujoco>
|
||||
<worldbody>
|
||||
<!-- Hand mocap bodies -->
|
||||
<body name="index-finger-tip-right" mocap="true"/>
|
||||
|
||||
<!-- Graspable object -->
|
||||
<body name="cube" pos="0 0 0.5">
|
||||
<freejoint/>
|
||||
<geom type="box" size="0.05 0.05 0.05" rgba="1 1 0 1"/>
|
||||
</body>
|
||||
|
||||
<!-- Equality constraint to connect finger to object -->
|
||||
<equality>
|
||||
<weld body1="index-finger-tip-right" body2="cube" active="false"/>
|
||||
</equality>
|
||||
</worldbody>
|
||||
|
||||
<actuator>
|
||||
<!-- Actuator to activate/deactivate weld constraint -->
|
||||
<general joint="cube" dyntype="none"/>
|
||||
</actuator>
|
||||
</mujoco>
|
||||
```
|
||||
|
||||
## Example: Pinch Detection
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
@app.add_handler("ON_MUJOCO_FRAME")
|
||||
async def detect_pinch(event: ClientEvent, sess: VuerSession):
|
||||
mocap_pos = event.value.get("mocap_pos")
|
||||
|
||||
if mocap_pos and len(mocap_pos) >= 10:
|
||||
# Get thumb tip and index finger tip positions
|
||||
thumb_tip = np.array(mocap_pos[4]) # Index 4
|
||||
index_tip = np.array(mocap_pos[9]) # Index 9
|
||||
|
||||
# Calculate distance
|
||||
distance = np.linalg.norm(thumb_tip - index_tip)
|
||||
|
||||
# Detect pinch
|
||||
if distance < 0.02: # 2cm threshold
|
||||
print("Pinch detected!")
|
||||
# Trigger grasp action
|
||||
```
|
||||
|
||||
## VR Access
|
||||
|
||||
1. Start server:
|
||||
```bash
|
||||
python your_script.py
|
||||
```
|
||||
|
||||
2. Set up ngrok:
|
||||
```bash
|
||||
ngrok http 8012
|
||||
```
|
||||
|
||||
3. Access via VR headset:
|
||||
```
|
||||
https://vuer.ai?ws=wss://xxxxx.ngrok.io
|
||||
```
|
||||
|
||||
4. Enable hand tracking in your VR headset settings
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use XR naming convention** - Follow the exact joint naming format
|
||||
2. **Define all needed mocap bodies** - Only tracked joints need mocap bodies
|
||||
3. **Set appropriate scale** - Scale simulation for VR comfort (e.g., 0.1)
|
||||
4. **Handle both hands** - Create separate mocap bodies for left and right
|
||||
5. **Test joint mapping** - Verify each joint is tracking correctly
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Hands not tracking
|
||||
- Verify SSL is properly set up
|
||||
- Check that hand tracking is enabled in VR headset
|
||||
- Confirm `Hands()` component is in the scene
|
||||
|
||||
### Mocap bodies not moving
|
||||
- Verify mocap body names match XR convention exactly
|
||||
- Check that `mocap="true"` is set in XML
|
||||
- Ensure body names include `-left` or `-right` suffix
|
||||
|
||||
### Poor tracking accuracy
|
||||
- Calibrate VR headset hand tracking
|
||||
- Ensure good lighting conditions
|
||||
- Check for hand occlusion issues
|
||||
|
||||
## Source
|
||||
|
||||
Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_hand_control.html
|
||||
Reference in New Issue
Block a user