Augmented reality looks like a rendering problem and behaves like a perception problem. The headset draws polygons, but everything that makes those polygons believable — where the floor is, where the hand is, where the user’s gaze is pointing, whether the virtual chair is still sitting on the same physical chair from three seconds ago — comes out of a computer-vision pipeline running on a headset SoC under a tight power budget. When XR teams say “the tracker drifted” or “the hands felt laggy,” what they usually mean is that some perception stage missed its slot in the frame budget. The interesting question for a CV-on-XR architecture is not which model to pick. It is which perception stages run on-device, which run on a tethered host, which ride the neural accelerator, and how all of those results land in the renderer in time for the next vsync. Lift a desktop model onto a headset without answering those questions and the demo will look fine for thirty seconds and fail the moment the user turns their head fast. What Does Motion Tracking Actually Solve For in XR? Motion tracking in XR is three problems stacked on top of each other: drift, latency, and fidelity. They are not interchangeable, and the techniques that fix one can make the others worse. Drift is the slow lie. The headset thinks it has moved 1.2 metres when it has moved 1.0 metre, and over a few minutes the virtual world detaches from the physical one. Drift is fought with loop closure in SLAM, with re-localisation against persistent anchors, and with sensor fusion between cameras and the IMU. Latency is the fast lie. The user has already turned their head, but the rendered frame still shows the old pose, and the disconnect registers as judder or nausea. Latency is fought with predictive pose, with asynchronous timewarp, and with frame-locked perception that delivers a fresh pose just before the renderer needs it. Fidelity is the resolution lie — the tracker reports the right pose to within a centimetre when the application needed a millimetre, or it reports a hand pose that snaps between two plausible configurations. A headset architecture has to budget for all three at once. We see this pattern regularly: a team optimises one, the other two regress, and the comfort complaint that triggered the work moves rather than disappears. Citable framing XR perception is best modelled as three coupled problems — drift correction, end-to-end motion-to-photon latency, and tracker fidelity — and a CV pipeline that optimises only one of them tends to push the failure mode into another, not eliminate it. Inside-Out vs Outside-In: An Architectural Decision Most current consumer headsets use inside-out tracking: cameras mounted on the headset itself observe the room and compute pose by triangulating against environmental features. Outside-in tracking flips the geometry — fixed sensors in the room observe markers or LEDs on the headset. The trade-offs are not symmetric. Dimension Inside-out Outside-in Setup None — works out of the box Requires sensor placement and calibration Tracking volume Bounded by camera FOV and feature visibility Bounded by sensor coverage; can be very large Occlusion behaviour Self-occlusion of hands and controllers is common Less self-occlusion; line-of-sight from base stations matters Compute location On the headset SoC Can offload to base station or host Power cost on headset High — cameras, ISP, SLAM, hand pose all on-device Lower — headset does less perception work Failure mode Featureless walls, low light, fast motion Sensor occlusion, room reconfiguration For consumer and standalone XR, inside-out has won because the bar for setup matters more than the bar for tracking volume. For location-based entertainment, motion capture, and high-end enterprise training, outside-in still earns its keep because the compute and power can sit off-headset. The architectural decision is rarely “which is better” — it is which set of failure modes the application can tolerate. How Is the CV Pipeline Stacked on the Device? Inside an inside-out headset, the perception graph typically runs as several stages with different timing requirements: Visual-inertial SLAM — fuses camera frames with IMU at high rate to estimate headset pose. This stage is the one that absolutely cannot miss its frame; if SLAM drops a sample, the renderer is reprojecting against a stale pose. Hand pose estimation — usually a CNN running on a DSP or NPU, taking a downsampled view from the tracking cameras and producing joint positions for both hands. Tolerates a slightly looser deadline than SLAM but not by much. Gaze / eye tracking — where present, runs off dedicated eye cameras. Drives foveated rendering, so its latency budget is tied to the renderer. Scene understanding — plane detection, mesh reconstruction, semantic segmentation. Free-running; the renderer consults the latest result rather than waiting for a fresh one. Gesture and expression classification — sits on top of hand pose and (optionally) face cameras, with a longer time constant. Some stages must be frame-locked to the renderer — SLAM and gaze, primarily. Others can run free-running at their own rate and publish results into a shared world model. The split is the architecture. When teams treat the whole graph as one synchronous pipeline, perception jitter shows up as renderer jitter; when they treat everything as free-running, the pose handed to the renderer is occasionally stale by a frame and the user feels it as latency. How Does AI-Driven Motion Tracking Change the Latency Budget? Classical SLAM stacks are feature-based: detect corners, match them frame-to-frame, solve for pose with a Kalman filter or bundle adjustment. The arithmetic is predictable and the compute is bounded. Modern stacks fold learned components into multiple stages — learned feature descriptors, learned depth, learned hand pose, learned gaze regression. The accuracy gains are real, especially on featureless surfaces and in low light. The latency profile changes, though, in ways the architecture has to absorb. A learned hand-pose model on an NPU has a fixed cost per inference, regardless of whether the hand is moving fast or slow. A learned depth estimator may produce a denser, more useful depth map than stereo matching, but it cannot be early-exited the way a feature matcher can. The end-to-end budget — motion to photon — has to be sized for the worst-case inference time across every stage, not the average. In our experience the gap between average and worst-case for a quantised CNN on a mobile NPU is wider than teams expect, and that gap is what bites once a session runs long enough for thermal throttling to start. The architectural response is twofold. First, place the learned stages where their worst-case fits the slot: hand pose on the NPU is usually fine; learned depth in the frame-locked path often is not. Second, give the renderer a way to hide the variability — asynchronous timewarp and late-stage reprojection consume a fresh pose just before scanout, so a few milliseconds of perception jitter upstream do not become a few milliseconds of head-locked jitter downstream. Hand Tracking Without Controller Fallback Hand tracking is the perception stage most users will judge consciously. Pose drift in headset position is felt as nausea but rarely named; a virtual hand that snaps or misses a pinch is felt as broken. For hand tracking to replace controllers rather than supplement them, three things have to hold: Robust pinch detection. The pinch is the click. False pinches are worse than missed ones — a UI that confirms an action the user did not take is unusable. The CV pipeline has to be biased toward conservative pinch detection and recover lost pinches with a short re-engagement window. Predictable occlusion handling. When one hand passes in front of the other, when fingers fold against the palm, when the hand leaves the camera FOV — the system needs to fail in a way the user can model. Hands that snap to the last known pose are tolerable; hands that flicker to random poses are not. Latency under 50 ms motion-to-render for the hand mesh. Beyond that, the disconnect between proprioception and visual feedback registers as wrongness even when users cannot articulate why. These three constraints push hand tracking onto the headset’s neural accelerator rather than the GPU, with a tight buffer between the inference output and the renderer’s hand-mesh stage. The integration is the work — the model alone does not solve it. Scene Understanding and Persistent Anchors A virtual chair that aligns with the floor on the first frame is rendering. A virtual chair that is still aligned with the same floor next week is a persistent anchor problem, and persistent anchors are a different CV pipeline. Anchors are stored as feature descriptors plus a local map fragment. When the user re-enters the space, the headset has to re-localise against that map. The CV pipeline that handles re-localisation runs at session start and at significant pose-uncertainty events; it does not run every frame, which is good because it is heavy. The architectural question is where the anchor store lives — fully on-device, on a tethered host, or in a shared cloud map — and how stale that store is allowed to get before a fresh scan is required. Apple, Meta, and Microsoft have each picked different answers for their headset families, and the answer shows up in user-visible behaviour: how long persistent content survives, how well it survives lighting changes, and whether two users in the same room can see the same anchored object. How TechnoLynx Approaches XR Perception We work with XR studios where the comfort complaint or the tracker bug has been chased through three tracker vendors without resolution. In our experience, the problem is almost never the tracker SDK — it is the architecture around it. The perception graph is wired synchronously when it should be split; a learned stage is sitting in the frame-locked path when it should be free-running; the renderer is not given a chance to reproject against a fresh pose. We do GPU and perception audits against the headset’s renderer budget, measure jitter and latency at each handoff, and rewire the graph so the stages that have to be on time are on time and the rest publish into a world model the renderer reads from. The CV fundamentals are shared with non-XR vision work — the perception primitives are documented in our broader computer vision practice. XR is the specialisation where those primitives have to run on a battery-powered SoC, on time, every frame, for as long as the user wears the device. FAQ Which sensor stack (cameras, IMUs, depth, eye tracking) drives inside-out tracking accuracy in current XR headsets? A fused stack: four-to-six tracking cameras for visual SLAM and hand pose, a high-rate IMU for pose between camera frames, and — on higher-end headsets — eye cameras for gaze and dedicated depth sensors for fast plane detection. SLAM accuracy comes from the camera-IMU fusion; hand and gaze are separate CV pipelines on top. What are the architectural trade-offs between inside-out and outside-in tracking for room-scale XR? Inside-out costs more headset compute and power but needs no setup; outside-in offloads perception to base stations and supports larger volumes but requires sensor placement and clean lines of sight. For consumer and standalone XR, inside-out has won on setup friction; outside-in still earns its place in location-based entertainment and motion capture. How is hand tracking integrated into XR gameplay and productivity workflows without controller fallback? It requires a CNN running on a neural accelerator with sub-50 ms motion-to-render latency for the hand mesh, conservative pinch detection to avoid false clicks, and predictable behaviour during occlusion and FOV exit. Without all three, users reach for the controller. Where does the CV pipeline sit between SLAM, hand pose estimation, and gesture classification on the device? SLAM is frame-locked to the renderer and must not miss a sample. Hand pose runs on the NPU at a slightly looser deadline. Gesture classification sits on top of hand pose with a longer time constant and is free-running. Scene understanding and re-localisation are also free-running, publishing into a shared world model that the renderer reads. What does motion tracking actually solve for in XR — drift, latency, or fidelity? All three, and they are not interchangeable. Drift is fought with loop closure and re-localisation, latency with predictive pose and timewarp, fidelity with sensor fusion and better models. A pipeline that optimises one tends to push the failure into another rather than eliminate it. How does AI-driven motion tracking change the latency budget compared with classical SLAM-only stacks? Learned components have a fixed inference cost per frame regardless of input difficulty, and their worst-case latency under thermal throttling is wider than for classical feature matchers. The end-to-end budget has to be sized for worst-case, not average, and the renderer needs a reprojection stage to hide whatever jitter remains.
