# MuJoCo VR Hand Control ## Overview This tutorial demonstrates how to control virtual hands in MuJoCo by leveraging mocap (motion capture) points that track user hand poses in VR environments. ## Important Requirement **SSL/HTTPS Required**: VR hand tracking requires secure connections. Use ngrok or localtunnel to set up SSL. See the [SSL Proxy WebXR tutorial](../basics/ssl-proxy-webxr.md) for setup instructions. ## Mocap Point API The implementation uses **XR Hand Naming Conventions** to link mocap bodies with hand joints. ### Naming Format ``` "{joint}-{left | right}" ``` Examples: - `wrist-right` - `middle-finger-phalanx-proximal-right` - `thumb-tip-left` ## Hand Joint Mapping The system defines **25 distinct hand joints** (indexed 0-24): ### Joint Index Reference ```python HAND_JOINTS = { 0: "wrist", # Thumb (1-4) 1: "thumb-metacarpal", 2: "thumb-phalanx-proximal", 3: "thumb-phalanx-distal", 4: "thumb-tip", # Index finger (5-9) 5: "index-finger-metacarpal", 6: "index-finger-phalanx-proximal", 7: "index-finger-phalanx-intermediate", 8: "index-finger-phalanx-distal", 9: "index-finger-tip", # Middle finger (10-14) 10: "middle-finger-metacarpal", 11: "middle-finger-phalanx-proximal", 12: "middle-finger-phalanx-intermediate", 13: "middle-finger-phalanx-distal", 14: "middle-finger-tip", # Ring finger (15-19) 15: "ring-finger-metacarpal", 16: "ring-finger-phalanx-proximal", 17: "ring-finger-phalanx-intermediate", 18: "ring-finger-phalanx-distal", 19: "ring-finger-tip", # Pinky finger (20-24) 20: "pinky-finger-metacarpal", 21: "pinky-finger-phalanx-proximal", 22: "pinky-finger-phalanx-intermediate", 23: "pinky-finger-phalanx-distal", 24: "pinky-finger-tip", } ``` ## Complete Example ```python import asyncio from vuer import Vuer, VuerSession from vuer.schemas import ( Scene, Fog, Sphere, MuJoCo, ContribLoader, Hands ) from vuer.events import ClientEvent app = Vuer() # Assets for hand simulation HAND_ASSETS = [ "/static/mujoco/hands/scene.xml", "/static/mujoco/hands/left_hand.xml", "/static/mujoco/hands/right_hand.xml", "/static/mujoco/hands/palm.obj", "/static/mujoco/hands/finger.obj", ] @app.add_handler("ON_MUJOCO_FRAME") async def on_mujoco_frame(event: ClientEvent, sess: VuerSession): """Handle physics updates""" print("ON_MUJOCO_FRAME", event.value) # Access mocap data mocap_pos = event.value.get("mocap_pos") mocap_quat = event.value.get("mocap_quat") # Process hand tracking data if mocap_pos and mocap_quat: # Update hand positions based on tracking pass @app.spawn(start=True) async def main(session: VuerSession): # Load MuJoCo library session.upsert @ ContribLoader( library="@vuer-ai/mujoco-ts", version="0.0.24", entry="dist/index.umd.js", key="mujoco-loader", ) await asyncio.sleep(2.0) # Set up scene with hands session.set @ Scene( bgChildren=[ # MuJoCo default styling Fog(color=0x2C3F57, near=10, far=20), # Add VR hands Hands(), # Background sphere Sphere( args=[50, 10, 10], materialType="basic", material=dict(color=0x2C3F57, side=1), ), ], # Initialize MuJoCo with hand model MuJoCo( src="/static/mujoco/hands/scene.xml", assets=HAND_ASSETS, scale=0.1, key="hand-sim", ), ) # Keep session alive while True: await asyncio.sleep(1.0) app.run() ``` ## Key Components ### Hands Component ```python Hands() ``` This enables VR hand tracking, providing position and orientation data for all 25 hand joints. ### Mocap Bodies in MuJoCo XML In your MuJoCo scene XML, define mocap bodies using the XR hand naming convention: ```xml ``` ## Accessing Hand Data ### Method 1: Event Handler ```python @app.add_handler("ON_MUJOCO_FRAME") async def on_frame(event: ClientEvent, sess: VuerSession): # Get mocap positions (3D coordinates) mocap_pos = event.value.get("mocap_pos") # Get mocap quaternions (orientations) mocap_quat = event.value.get("mocap_quat") if mocap_pos: # mocap_pos is a list of [x, y, z] positions # Order matches the mocap body order in XML wrist_pos = mocap_pos[0] thumb_tip_pos = mocap_pos[1] # etc. print(f"Wrist position: {wrist_pos}") ``` ### Method 2: Direct Hand Tracking ```python from vuer.events import ClientEvent @app.add_handler("HAND_MOVE") async def on_hand_move(event: ClientEvent, sess: VuerSession): """Handle hand tracking events directly""" hand_data = event.value # Access hand side side = hand_data.get("side") # "left" or "right" # Access joint positions joints = hand_data.get("joints") # List of 25 joint positions print(f"{side} hand moved") print(f"Wrist: {joints[0]}") print(f"Index tip: {joints[9]}") ``` ## Creating Hand-Object Interactions ```python # In your MuJoCo XML ``` ## Example: Pinch Detection ```python import numpy as np @app.add_handler("ON_MUJOCO_FRAME") async def detect_pinch(event: ClientEvent, sess: VuerSession): mocap_pos = event.value.get("mocap_pos") if mocap_pos and len(mocap_pos) >= 10: # Get thumb tip and index finger tip positions thumb_tip = np.array(mocap_pos[4]) # Index 4 index_tip = np.array(mocap_pos[9]) # Index 9 # Calculate distance distance = np.linalg.norm(thumb_tip - index_tip) # Detect pinch if distance < 0.02: # 2cm threshold print("Pinch detected!") # Trigger grasp action ``` ## VR Access 1. Start server: ```bash python your_script.py ``` 2. Set up ngrok: ```bash ngrok http 8012 ``` 3. Access via VR headset: ``` https://vuer.ai?ws=wss://xxxxx.ngrok.io ``` 4. Enable hand tracking in your VR headset settings ## Best Practices 1. **Use XR naming convention** - Follow the exact joint naming format 2. **Define all needed mocap bodies** - Only tracked joints need mocap bodies 3. **Set appropriate scale** - Scale simulation for VR comfort (e.g., 0.1) 4. **Handle both hands** - Create separate mocap bodies for left and right 5. **Test joint mapping** - Verify each joint is tracking correctly ## Troubleshooting ### Hands not tracking - Verify SSL is properly set up - Check that hand tracking is enabled in VR headset - Confirm `Hands()` component is in the scene ### Mocap bodies not moving - Verify mocap body names match XR convention exactly - Check that `mocap="true"` is set in XML - Ensure body names include `-left` or `-right` suffix ### Poor tracking accuracy - Calibrate VR headset hand tracking - Ensure good lighting conditions - Check for hand occlusion issues ## Source Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_hand_control.html