zhongwei/gh-vuer-ai-vuer-skill-marketplace-skills-docs-vuer-ai

Files

Zhongwei Li 265175ed82 Initial commit

2025-11-30 09:05:02 +08:00

8.1 KiB

Raw Blame History

MuJoCo VR Hand Control

Overview

This tutorial demonstrates how to control virtual hands in MuJoCo by leveraging mocap (motion capture) points that track user hand poses in VR environments.

Important Requirement

SSL/HTTPS Required: VR hand tracking requires secure connections. Use ngrok or localtunnel to set up SSL.

See the SSL Proxy WebXR tutorial for setup instructions.

Mocap Point API

The implementation uses XR Hand Naming Conventions to link mocap bodies with hand joints.

Naming Format

"{joint}-{left | right}"

Examples:

wrist-right
middle-finger-phalanx-proximal-right
thumb-tip-left

Hand Joint Mapping

The system defines 25 distinct hand joints (indexed 0-24):

Joint Index Reference

HAND_JOINTS = {
    0: "wrist",
    # Thumb (1-4)
    1: "thumb-metacarpal",
    2: "thumb-phalanx-proximal",
    3: "thumb-phalanx-distal",
    4: "thumb-tip",
    # Index finger (5-9)
    5: "index-finger-metacarpal",
    6: "index-finger-phalanx-proximal",
    7: "index-finger-phalanx-intermediate",
    8: "index-finger-phalanx-distal",
    9: "index-finger-tip",
    # Middle finger (10-14)
    10: "middle-finger-metacarpal",
    11: "middle-finger-phalanx-proximal",
    12: "middle-finger-phalanx-intermediate",
    13: "middle-finger-phalanx-distal",
    14: "middle-finger-tip",
    # Ring finger (15-19)
    15: "ring-finger-metacarpal",
    16: "ring-finger-phalanx-proximal",
    17: "ring-finger-phalanx-intermediate",
    18: "ring-finger-phalanx-distal",
    19: "ring-finger-tip",
    # Pinky finger (20-24)
    20: "pinky-finger-metacarpal",
    21: "pinky-finger-phalanx-proximal",
    22: "pinky-finger-phalanx-intermediate",
    23: "pinky-finger-phalanx-distal",
    24: "pinky-finger-tip",
}

Complete Example

import asyncio
from vuer import Vuer, VuerSession
from vuer.schemas import (
    Scene, Fog, Sphere,
    MuJoCo, ContribLoader,
    Hands
)
from vuer.events import ClientEvent

app = Vuer()

# Assets for hand simulation
HAND_ASSETS = [
    "/static/mujoco/hands/scene.xml",
    "/static/mujoco/hands/left_hand.xml",
    "/static/mujoco/hands/right_hand.xml",
    "/static/mujoco/hands/palm.obj",
    "/static/mujoco/hands/finger.obj",
]

@app.add_handler("ON_MUJOCO_FRAME")
async def on_mujoco_frame(event: ClientEvent, sess: VuerSession):
    """Handle physics updates"""
    print("ON_MUJOCO_FRAME", event.value)

    # Access mocap data
    mocap_pos = event.value.get("mocap_pos")
    mocap_quat = event.value.get("mocap_quat")

    # Process hand tracking data
    if mocap_pos and mocap_quat:
        # Update hand positions based on tracking
        pass

@app.spawn(start=True)
async def main(session: VuerSession):
    # Load MuJoCo library
    session.upsert @ ContribLoader(
        library="@vuer-ai/mujoco-ts",
        version="0.0.24",
        entry="dist/index.umd.js",
        key="mujoco-loader",
    )

    await asyncio.sleep(2.0)

    # Set up scene with hands
    session.set @ Scene(
        bgChildren=[
            # MuJoCo default styling
            Fog(color=0x2C3F57, near=10, far=20),

            # Add VR hands
            Hands(),

            # Background sphere
            Sphere(
                args=[50, 10, 10],
                materialType="basic",
                material=dict(color=0x2C3F57, side=1),
            ),
        ],

        # Initialize MuJoCo with hand model
        MuJoCo(
            src="/static/mujoco/hands/scene.xml",
            assets=HAND_ASSETS,
            scale=0.1,
            key="hand-sim",
        ),
    )

    # Keep session alive
    while True:
        await asyncio.sleep(1.0)

app.run()

Key Components

Hands Component

Hands()

This enables VR hand tracking, providing position and orientation data for all 25 hand joints.

Mocap Bodies in MuJoCo XML

In your MuJoCo scene XML, define mocap bodies using the XR hand naming convention:

<mujoco>
  <worldbody>
    <!-- Right hand mocap bodies -->
    <body name="wrist-right" mocap="true">
      <geom type="sphere" size="0.02" rgba="1 0 0 0.5"/>
    </body>

    <body name="thumb-tip-right" mocap="true">
      <geom type="sphere" size="0.01" rgba="0 1 0 0.5"/>
    </body>

    <body name="index-finger-tip-right" mocap="true">
      <geom type="sphere" size="0.01" rgba="0 0 1 0.5"/>
    </body>

    <!-- Add more joints as needed -->

    <!-- Left hand mocap bodies -->
    <body name="wrist-left" mocap="true">
      <geom type="sphere" size="0.02" rgba="1 0 0 0.5"/>
    </body>

    <!-- Add left hand joints -->
  </worldbody>
</mujoco>

Accessing Hand Data

Method 1: Event Handler

@app.add_handler("ON_MUJOCO_FRAME")
async def on_frame(event: ClientEvent, sess: VuerSession):
    # Get mocap positions (3D coordinates)
    mocap_pos = event.value.get("mocap_pos")

    # Get mocap quaternions (orientations)
    mocap_quat = event.value.get("mocap_quat")

    if mocap_pos:
        # mocap_pos is a list of [x, y, z] positions
        # Order matches the mocap body order in XML
        wrist_pos = mocap_pos[0]
        thumb_tip_pos = mocap_pos[1]
        # etc.

        print(f"Wrist position: {wrist_pos}")

Method 2: Direct Hand Tracking

from vuer.events import ClientEvent

@app.add_handler("HAND_MOVE")
async def on_hand_move(event: ClientEvent, sess: VuerSession):
    """Handle hand tracking events directly"""
    hand_data = event.value

    # Access hand side
    side = hand_data.get("side")  # "left" or "right"

    # Access joint positions
    joints = hand_data.get("joints")  # List of 25 joint positions

    print(f"{side} hand moved")
    print(f"Wrist: {joints[0]}")
    print(f"Index tip: {joints[9]}")

Creating Hand-Object Interactions

# In your MuJoCo XML
<mujoco>
  <worldbody>
    <!-- Hand mocap bodies -->
    <body name="index-finger-tip-right" mocap="true"/>

    <!-- Graspable object -->
    <body name="cube" pos="0 0 0.5">
      <freejoint/>
      <geom type="box" size="0.05 0.05 0.05" rgba="1 1 0 1"/>
    </body>

    <!-- Equality constraint to connect finger to object -->
    <equality>
      <weld body1="index-finger-tip-right" body2="cube" active="false"/>
    </equality>
  </worldbody>

  <actuator>
    <!-- Actuator to activate/deactivate weld constraint -->
    <general joint="cube" dyntype="none"/>
  </actuator>
</mujoco>

Example: Pinch Detection

import numpy as np

@app.add_handler("ON_MUJOCO_FRAME")
async def detect_pinch(event: ClientEvent, sess: VuerSession):
    mocap_pos = event.value.get("mocap_pos")

    if mocap_pos and len(mocap_pos) >= 10:
        # Get thumb tip and index finger tip positions
        thumb_tip = np.array(mocap_pos[4])   # Index 4
        index_tip = np.array(mocap_pos[9])   # Index 9

        # Calculate distance
        distance = np.linalg.norm(thumb_tip - index_tip)

        # Detect pinch
        if distance < 0.02:  # 2cm threshold
            print("Pinch detected!")
            # Trigger grasp action

VR Access

Start server:

python your_script.py

Set up ngrok:

ngrok http 8012

Access via VR headset:

https://vuer.ai?ws=wss://xxxxx.ngrok.io

Enable hand tracking in your VR headset settings

Best Practices

Use XR naming convention - Follow the exact joint naming format
Define all needed mocap bodies - Only tracked joints need mocap bodies
Set appropriate scale - Scale simulation for VR comfort (e.g., 0.1)
Handle both hands - Create separate mocap bodies for left and right
Test joint mapping - Verify each joint is tracking correctly

Troubleshooting

Hands not tracking

Verify SSL is properly set up
Check that hand tracking is enabled in VR headset
Confirm Hands() component is in the scene

Mocap bodies not moving

Verify mocap body names match XR convention exactly
Check that mocap="true" is set in XML
Ensure body names include -left or -right suffix

Poor tracking accuracy

Calibrate VR headset hand tracking
Ensure good lighting conditions
Check for hand occlusion issues

Source

Documentation: https://docs.vuer.ai/en/latest/tutorials/physics/mocap_hand_control.html

8.1 KiB Raw Blame History