TL;DR:

  • Meta Quest 3/3S and Apple Vision Pro both support full hand tracking without physical controllers in 2026; enterprise AR headsets (HoloLens 2, Magic Leap 2) have supported it longer
  • Apple visionOS uses hands as the only primary input method (gaze + pinch gesture); Meta’s platform supports hands alongside controllers as an alternative mode
  • The interaction primitives that work well are pinch-to-select, grab, and proximity-based menus; the ones that don’t work well are fine motor tasks, typing, and anything requiring sustained precision
  • For developers: Meta’s Hand Tracking SDK and visionOS’s RealityKit/SwiftUI spatial APIs both provide 26-joint skeletal hand data; OpenXR’s hand tracking extension unifies the model across platforms

The promise of hand tracking — just reach out and interact with virtual content directly — has been in XR demos for a decade. The gap between demo and usable product has been persistent: tracking breaks in low light, gestures misfire, latency makes fine manipulation feel wrong, and users fatigue faster than they would with physical controllers that provide haptic resistance.

In 2026, that gap has narrowed substantially. Not closed — but narrowed enough that hands-only XR is genuinely appropriate for a growing class of applications, and understanding where it works (and where it doesn’t) has become a practical development decision rather than a research question.

The Hardware Landscape

Meta Quest 3 and Quest 3S: Meta’s current consumer headsets use infrared cameras on the headset’s outer shell to track hands at up to 90 frames per second. The tracking works at roughly arm’s length and handles most natural hand positions. It loses fidelity in direct sunlight (IR interference) and struggles with very fast movements. Meta’s hand tracking has improved substantially through software updates since the Quest 2 days — the tracking loss scenarios that used to make hands-only apps frustrating have been largely resolved for typical indoor use.

Apple Vision Pro: visionOS treats eye tracking + hand gestures as the primary input model. There are no controllers — the hands are the interface, full stop. Apple’s approach is intentionally limited in gesture vocabulary: the main interaction is a pinch (thumb and index finger touching) triggered while looking at the target element. This minimalism sidesteps the gesture recognition problem by using only a small set of highly reliable motions. The result is consistently accurate and low-fatigue, but constrains what interaction models are feasible.

HoloLens 2 and Magic Leap 2: Enterprise AR headsets have supported hand tracking as a primary input for several years. HoloLens 2’s hand tracking model provides 26 joints per hand and has been production-tested across manufacturing, surgery assistance, and field service applications. Magic Leap 2 adds eye tracking to complement hand input.

Samsung Galaxy XR: The Android XR platform that launched in 2026 supports hand tracking with similar capabilities to Quest, using Google’s MediaPipe hand tracking pipeline adapted for the headset camera configuration.

How Skeletal Hand Tracking Works

At the API level, hand tracking exposes the 3D position and orientation of up to 26 joints per hand — fingertips, knuckle joints, the wrist, and the palm. At 90fps, this gives you a full skeletal model of each hand’s pose in world space on every frame.

Applications use this data in a few ways:

Pose recognition: Match the current joint configuration against a library of known poses (“pinch”, “grab”, “open palm”, “point”). This is how gesture-based UI triggers work.

Physics simulation: Treat the hand as a collider in the scene physics simulation. Virtual objects respond to hand contact, slide, or push as if there were physical resistance.

Ray casting: Project a ray from the index fingertip (or the wrist forward vector) and detect intersections with UI panels in the scene. This mirrors how laser pointer interaction works with controllers, but uses finger pointing as the input mechanism.

In OpenXR, hand tracking data is exposed via the XR_EXT_hand_tracking extension, which provides joint poses in a standardised format across platforms:

// OpenXR hand tracking joint query
XrHandJointLocationEXT jointLocations[XR_HAND_JOINT_COUNT_EXT];
XrHandJointLocationsEXT handJointLocations{
    XR_TYPE_HAND_JOINT_LOCATIONS_EXT,
    nullptr,
    XR_TRUE,
    XR_HAND_JOINT_COUNT_EXT,
    jointLocations
};

xrLocateHandJointsEXT(handTracker, &locateInfo, &handJointLocations);

// Index fingertip is joint index 8
XrPosef fingertipPose = jointLocations[XR_HAND_JOINT_INDEX_TIP_EXT].pose;

For visionOS, spatial interactions are abstracted higher — RealityKit’s TapGesture and DragGesture handle the gaze-plus-pinch model without exposing raw joint data for most use cases. Raw hand anchors are available via the ARKit hand tracking anchor API for applications that need skeletal data directly.

Interaction Patterns That Work

Experience with hand tracking in production applications has converged on a set of patterns that hold up reliably:

Near-field manipulation: Picking up, moving, and placing virtual objects works well when interactions happen within 60cm of the body. The closer the hands are to the body, the more stable the tracking and the less fatigue users experience.

Pinch selection on large targets: Menus, cards, and buttons that are at least 4–5cm in the virtual environment are reliably selectable with a pinch gesture. Smaller targets increase error rates significantly.

Gaze-directed interaction (visionOS model): Looking at a target, then pinching to confirm — without requiring the hand to physically point at the target — is accurate and low-fatigue for extended sessions. The cognitive separation of “where I’m looking” from “what my hands are doing” takes brief habituation but becomes natural.

Proximity menus: Menus that appear near the user’s palm when the palm faces upward (“palm menu” pattern) are effective for frequently accessed controls. They don’t require the user to look at their hand to activate.

Where Controllers Are Still Better

Sustained precision: Fine motor tasks — 3D modelling, surgical simulation, any interaction requiring repeated sub-centimetre accuracy — are better with physical controllers that provide haptic resistance and a stable grip surface. Hands fatigue in unsupported mid-air positions in ways that controllers held against the body do not.

Gaming: Button mashing, trigger holds, and thumbstick navigation have no good hand tracking equivalents. Controllers remain the right input for games that evolved around their affordances.

Text input: Virtual keyboards with hand tracking are slow and error-prone compared to physical keyboards, voice input, or controller-based typing interfaces. The absence of tactile feedback makes touch-typing impossible.

Low-light or outdoor environments: IR-based hand tracking degrades significantly in direct sunlight or very low light. Applications that need to work reliably across varied environments should plan for controller fallback.

Development Considerations

If you’re building for hand tracking as a primary interaction mode:

  • Design for near-field interactions first; far-field ray casting is less reliable and more fatiguing
  • Use gesture confirmation (two-step: hover then pinch) rather than hover-to-activate for destructive actions
  • Test with users who have rings, nail polish, and dark skin tones — tracking systems have historically performed less consistently across these variables, though recent models have improved
  • Provide clear visual feedback when the system has detected a valid gesture pose; users lose confidence when they can’t tell if the system saw their input
  • Target Meta Quest first for consumer deployments, visionOS for high-end enterprise or productivity scenarios, HoloLens 2 for existing enterprise AR deployments in manufacturing and field service

The use cases where hands-only input is already demonstrably better than controllers: guided procedures (surgery, manufacturing assembly steps where the worker’s hands are occupied), kiosk-style interactions where users shouldn’t need to hold a controller, and passive experiences where users want to reach out and interact naturally without learning a controller mapping. That’s a meaningful and growing set of applications, even before the technology improves further.