Yes, virtual reality uses artificial intelligence, including deep neural networks for perception tasks such as eye tracking and gesture recognition. AI and extended reality meet in a few jobs that never stop running: working out where your head is from cameras and inertial sensors, following your hands and eyes, and building a model of the room. XR workloads have to respond within tens of milliseconds on a device with tight compute and battery limits. The heaviest steps, such as bundle adjustment, object detection or the whole visual-inertial odometry stage, can be offloaded to an edge server when the network can carry the traffic.
One distinction matters before going further. Head tracking in ILLIXR uses visual-inertial odometry, while its eye tracking and XRBench’s hand tracking are neural networks. That mix makes AI on a headset a systems problem rather than a model problem.
Does VR use AI, and where on the headset does it run?
The ILLIXR runtime splits XR work into three pipelines: perception, visual and audio. In ILLIXR, eye tracking, the neural-network component, sits among the perception components. The visual pipeline’s application depends on poses from perception, and its reprojection step also corrects lens distortion. In the 3GPP model of an XR engine, application-level AI is a separate module, specific to each application, alongside physics, audio and localisation.
In the open-source ILLIXR system, visual-inertial odometry ran on the CPU while eye tracking and scene reconstruction ran on the GPU. Commercial chips add dedicated blocks: Qualcomm describes the Snapdragon XR2 Gen 2 as having a hardware-accelerated computer vision block for low-power 6DoF tracking and support for 10 concurrent cameras covering eye, hand, head and face tracking plus depth sensing. Apple lists six world-facing tracking cameras and four eye-tracking cameras on Vision Pro.
Head, hand and eye: how AI motion tracking works on the headset
The table summarises what the sources describe.
| Perception task | Typical method | Where it usually runs | What drives its budget |
|---|---|---|---|
| Head tracking (6DoF pose) | Visual-inertial odometry or SLAM, plus an IMU integrator | On-device CPU or DSP; VIO can be offloaded | Pose queries at 120 to 144 Hz for reprojection |
| Hand tracking | Cascaded hand detection then tracking, or a single end-to-end multi-view network | On-device | About 10 inferences per second minimum for detection, as assumed in one memory design study |
| Eye tracking | Segmentation networks such as RITnet, then gaze estimation | On-device GPU or accelerator | About 0.1 inferences per second minimum for segmentation, as assumed in one memory design study |
| Scene reconstruction | Dense SLAM such as ElasticFusion | On-device GPU, or offloaded | Map updates from colour and depth data every frame |
| Object detection | CNN detectors | Often on an edge server, with local tracking | Over 50 ms per HD frame even on a Tegra TX2-class mobile GPU |
Head tracking: fast integration, slow correction
Head tracking pairs two estimators because neither works alone. Visual-inertial odometry is accurate but runs only at camera rates of tens of hertz, while IMU integration runs at hundreds of hertz but drifts without correction. The runtime adds the IMU-integrated relative motion to the latest VIO pose to answer each high-rate pose query. Meta’s Oculus Insight goes further, using low-latency IMU data and a kinematic model that predicts where the user’s head will be, so that the apparent latency disappears. Head tracking is also expensive: prior work cited by the RemoteVIO authors puts it at 35% to 55% of all CPU cycles in an XR system.
Hands and eyes: cascades of small networks
Hand interaction is often built from a detection model that triggers a tracking model, and when no hand is found the tracking model does not run at all. That kind of dynamic, cascaded workload makes scheduling harder than running a single network at a fixed rate. Meta’s UmeTrack work moved hand tracking towards a single network trained end-to-end on the multi-view images from the headset’s wide-angle cameras, predicting absolute hand pose directly. For architectural patterns, including how hand-tracking output reaches applications, our piece on architectural patterns for AI motion tracking in XR goes deeper.
How do XR headsets fit AI inside a 20 ms latency budget and a sub-watt power budget?
The work reviewed here attacks both budgets through lower precision, specialised accelerators and offloading. The ILLIXR researchers use a motion-to-photon target of 20 ms for VR and 5 ms for AR. Motion-to-photon latency is the age of the pose behind the image when its pixels start to appear, made up of IMU sample age, reprojection time and swap wait. The power side is just as strict: for some vision use cases, high resolution and frame rates above 90 Hz are needed within power budgets under 1 W, and wearable thermals and battery life are first-order design limits.
On an NVIDIA Jetson AGX Xavier configured to approximate commercial mobile XR devices, neither the high-performance nor the low-power configuration met the AR motion-to-photon target for an average frame. In the low-power configuration only the audio pipeline met its frame-rate target. Even that platform remained two orders of magnitude away from the ideal power figure. Power also hides in unexpected places: on the low-power Jetson, SoC and system components (display, storage, I/O) used more than half of total power.
Quantisation buys memory, and costs some accuracy
The XR-NPE authors report that off-chip data movement accounts for almost 60% of energy consumption and latency at system level, and a design-space study of hand detection and eye segmentation found memory power far more significant than compute on systolic accelerators. Shrinking weights shrinks that traffic. In the XR-NPE work, a visual-inertial odometry model fell from 13.5 MB at FP32 to 2.42 MB with mixed FP4 and Posit precision, with FP4 adding 0.72 percentage points of translation RMSE over FP32. The caveat is real: the same paper notes that existing VIO models are highly sensitive to quantisation, with accuracy losses of up to 83% against an FP32 baseline, and it uses quantisation-aware training while keeping sensitive layers at higher precision. INT8 is the more conventional step, applied with NVIDIA TensorRT to hand and eye models in one study and giving a 3 to 4 times latency improvement for an edge detection model in another.
Accelerators, and how many of them
Specialised hardware is the second lever, and its sizing changes the results. A dedicated VIO accelerator took up to 72% of the area in the Quest 2 perception design, according to the XR-NPE authors. In XRBench simulations of 8-bit models, an accelerator with 4K processing elements dropped 47.1% of frames while an 8K design dropped 2.3%. The same benchmark found that the best accelerator style differed by usage scenario, so XR systems need co-design with their scenarios. On the software side, Oculus Insight splits mapping, tracking and IMU processing into separate threads, runs map updates asynchronously on the DSP and gives IMU data a higher-priority thread writing into shared memory.
What stays on the headset, and what can go to the edge?
The rule that falls out of the evidence: anything on the critical path to the next displayed frame stays local, and anything heavy and tolerant of delay is a candidate for offload. The 3GPP report on XR over 5G states that spatial mapping and localisation can run on the device, with network elements supporting them in several ways. The split point decides how much data the link has to carry.
Worked example: bandwidth at each split point in a SLAM pipeline
One headset tracking study calculated how much data each split point would send, using stereo 2048 × 732 grayscale images at 10 Hz. The figures below are theirs; the comparison is the point.
| What the headset sends | Data rate required |
|---|---|
| Raw stereo images (offload everything) | 240 Mbps |
| MPEG-compressed video | 22.59 Mbps |
| SIFT features, about 1,000 per frame | 10.72 Mbps |
| ORB features, about 1,000 per frame | 3.04 Mbps |
| Map matches, about 300 per frame | 672 kbps |
| Pose only | 35.84 kbps |
The split works because the expensive step, bundle adjustment, runs in the cloud. In that pipeline, bundle adjustment took 97.9% of SLAM processing time on a CPU, while feature extraction and stereo matching took under 1%. Extracting features locally and shipping sparse descriptors lets the edge do the expensive part for a fraction of the raw-image bandwidth.
Offloading head tracking itself
RemoteVIO moved visual-inertial odometry to a remote server and kept IMU integration on the device, which keeps producing poses at IMU rate while the next VIO correction is in flight. Measured on Jetson hardware, this cut CPU power by up to 52% (42% on average) and full-system power by up to 20% (13% on average). Compression and radio added cost, but about 0.4 W extra on 5G against total savings of 2.2 W. User experience held under typical network conditions and often degraded once round-trip time went above 200 ms, with jitter the most common complaint.
Offloading detection, tracking locally
Object detection follows a different offloading pattern. In one edge-offloading system, CNN detection ran on an edge server while the headset tracked previously detected objects using motion vectors from its own video encoder, taking 2.24 ms per frame on a Jetson TX2. With rendering decoupled from the offload pipeline, the system stayed within the 16.7 ms frame time at 60 fps.
The energy and latency trade-off is not fixed
Offloading trades local compute energy for transmission energy and network delay. In a study of deep-reinforcement-learning-driven edge offloading for XR pipelines, fully local execution drew 20.8 W against 7.6 W for full offload, but fixed offloading lost more latency compliance when bandwidth varied than an adaptive policy. In one DNN partitioning study, the policy shifted work to local computing as maximum transmit power rose, and to the edge as edge capacity grew. A static split chosen in the lab is a guess about the network your users will have.
Where visual tracking breaks: low light and low texture
Low illumination, reduced contrast, sensor noise and motion blur degrade both feature extraction and matching. In a 2026 benchmark of six SLAM and odometry systems on low-light sequences, only Kimera-VIO tracked all five sequences to completion, while classical monocular pipelines and the filter-based OpenVINS failed or diverged on most of the harder ones. The authors conclude that RGB-only SLAM stays stable in low light only when inertial fusion and global optimisation are both present, because the IMU carries short-term motion estimation through blur, glare or darkness. That benchmark targets ground-vehicle navigation rather than headsets, so we read it as a warning about the failure mode. For more on the camera side of this, see our overview of computer vision in virtual and augmented reality.
Moving a perception pipeline to the target hardware is engineering work
A tracking model that works on a workstation is not yet a tracking model that works on a headset. The ILLIXR team describe tuning XR component parameters for deployed hardware as a manual, mostly ad hoc process involving many quality and system metrics at once. They also argue that a unique accelerator per task is likely impractical given the number of tasks and the power and area limits, so algorithms have to be mapped onto shared, programmable hardware with its own software stack. Meta solved part of this for Quest by building a model that simulated the mobile chipset’s performance on data-centre servers, so algorithm changes were evaluated against the headset’s actual constraints.
TechnoLynx’s AI Porting & Deployment Pack gets perception workloads running on edge boards and embedded targets, starting with feasibility and then porting one workload with a reproducible runbook.
This work is most valuable when scoped against a measurable trigger: a performance ceiling, a deployment-target constraint or a lock-in cost. Rendering sits outside this page, and our piece on GPU optimisation for XR performance covers that side. Perception teams still have to settle how much accuracy each stage can give up before the wearer notices, and the RemoteVIO study found standard quantitative metrics were not well correlated with user experience.