The first fully functioning head-mounted VR system was built in 1968, when Ivan Sutherland and Bob Sproull made a tracked, binocular head-mounted display. Across every era since, the core loop has stayed the same: sense the user, render the result and return it to the eye with low enough latency. Today a key GPU lever on that loop is foveated rendering, which concentrates detail where the eye is looking. The other lever is deciding which work stays on the headset and which gets pre-rendered elsewhere.
Who invented virtual reality, and what was the first headset?
IEEE and ACM SIGGRAPH both point to Ivan Sutherland, and IEEE describes his 1968 head-mounted display as an early precursor to modern VR systems. The groundwork came earlier. In 1963 his Sketchpad paired a vector display with a tracked light pen, and two years later he described the “ultimate display” as a room in which the computer could control the existence of matter. The 1968 headset, built with Bob Sproull, redrew its imagery as the wearer’s head moved and turned. It was tracked by a counterbalanced mechanical arm and an ultrasonic system. ACM SIGGRAPH’s history archive calls it the first fully functioning 3D AR/VR system: real-time, see-through, stereo and perspective, with two head trackers, ultrasonic and mechanical (the latter known as the “Sword of Damocles”).
“First headset” depends on what you count. The Sutherland system is a useful starting point because it established the enduring objective of making a computed world respond coherently to the viewer’s actions.
Five stages of VR evolution, from lab rig to standalone headset
An arXiv chronology argues that VR did not develop as one line of devices. It developed as the integration of stereo display, head and body tracking, real-time graphics, spatial input and feedback into one stable perception–action loop. The same chronology splits the period after the lab era into five stages that overlap rather than replace each other:
| Stage | Years | What changed technically |
|---|---|---|
| Assembled access | 1991–2006 | Researchers and public venues combined specialised and consumer parts to get below laboratory cost. |
| Commodity sensing and mobile displays | 2007–2012 | Game controllers, depth cameras, smartphones and cheap viewers became reusable building blocks. |
| Packaged consumer platforms | 2013–2018 | Phone-slot viewers, PC headsets, room-scale tracking and early standalone systems turned prototypes into supported products. |
| Standalone ecosystems | 2019–2023 | Inside-out tracking, hand sensing, app stores and interoperability cut setup time, at the cost of more dependence on the platform. |
| Spatial-platform convergence | 2024–2026 | Passthrough mixed reality, eye and hand input and learned computation pulled VR, AR and everyday computing into one platform layer. |
The table shows where the work moved. External computers, base stations and controllers gradually gave way to onboard visual–inertial tracking, optical hand tracking and standalone compute. Perceptual rendering shifted part of the performance problem away from brute-force graphics and towards models of gaze and attention.
When did VR become popular, and what made it possible?
Consumer VR took off around 2012. The Furion authors date “the consumer virtual reality revolution” to Palmer Luckey’s Kickstarter campaign for the Oculus Rift that year, and Facebook bought Oculus in 2014. In June 2012 John Carmack showed a Doom demo on Luckey’s prototype at E3. According to the Rift’s tracking engineers, that demo convinced the industry that low-cost, high-fidelity VR could be built from MEMS sensors and display technology taken from the smartphone industry.
The Rift team had to fight latency, which they described as the main cause of VR simulator sickness up to that point, using nothing more than a simple MEMS inertial measurement unit. They used predictive tracking to reduce effective latency. The filter estimates where the head will be when the frame reaches the retina, not where it was when the sensor last sampled. The gyros reported angular velocity at 1000 Hz, and the team found that the predictive filters needed only a few milliseconds of data and performed well up to 50 ms ahead. (Our piece on motion sensors in AR and VR systems covers the sensing side in more depth.)
The products that followed removed the setup work one step at a time:
- HTC Vive shipped Lighthouse tracking and controllers for room-scale, six-degree-of-freedom PC VR.
- Oculus Go put display, compute, tracking, audio and software distribution into one self-contained US$199 device.
- On 21 May 2019, Oculus Quest and Rift S shipped with camera-based Insight tracking. Quest delivered room-scale head and controller tracking with no PC and no base stations.
- Quest 2 shipped in 2020 at US$299 with higher resolution and more processing power, which sped up adoption of standalone six-degree-of-freedom VR.
Quest removed the external tracking cameras. Meta describes it as the first consumer VR device with full six-degree-of-freedom tracking and dual-controller tracking built into the headset. That tracking runs on SLAM: computer vision fuses data from several sensors to fix the headset’s position inside a continuously updated map. The team’s main challenge was making that tracking precise and also efficient enough to run on a mobile device.
Motion-to-photon latency: the budget that never relaxed
Motion-to-photon latency is the time between a head movement and the matching change on the retina. The Rift paper counts sensing, USB transfer, sensor fusion, game simulation, rendering and video output in that pipeline, plus pixel switching times and any accidental frame buffering. In 2014 the same paper called 60 ms a commonly cited upper bound for an acceptable experience, said the ideal was below 20 ms, and put the modern interval at 30 to 50 ms.
Sources do not agree on a single threshold. The Furion paper gives under 25 ms for VR, against 100–200 ms for ordinary computer games, because the display sits so close to the eye. A 2025 paper on wireless VR sets the bar at 14 ms or less. It splits that budget into sensing (0.5–2 ms), input processing (1–3 ms), rendering (2–10 ms) and wireless frame transport (5–10 ms or more).
Frame rate is only one component of that budget. The HTC Vive used in NVIDIA’s 2017 foveation study took about 11.11 ms from synchronised transmission to pixel flash. Still, the arXiv chronology notes that no single refresh-rate threshold defines comfort: tracking quality, prediction, content motion and locomotion design all count alongside display rate. Our deep dive into VR technology covers the wider hardware and software picture.
Foveated rendering: why it became a key GPU lever
Foveated rendering exploits the fact that visual acuity falls off away from the centre of gaze. It shows a high-quality image in central (foveal) vision and a cheaper one in the periphery, which cuts compute cost. Peripheral pixels are the ones the viewer resolves least, so they are the cheapest place to save work.
Fixed versus eye-tracked foveation
There are two kinds: fixed and eye-tracked. Fixed foveated rendering (FFR) does not use eye tracking: the high-resolution region stays in the centre of the frame. Eye-tracked (dynamic) foveation moves the high-resolution region to wherever the eye is looking, so it needs eye-tracking hardware. Meta’s documentation names Meta Quest Pro and Meta VR Glasses as having that eye-tracking hardware.
On Quest headsets, FFR works through tiled rendering. The GPU splits each frame into dozens of tiles and renders tiles near the edge of the eye buffer at lower resolution. Meta says FFR can deliver up to a 25% performance gain in pixel-intensive applications, while apps with very simple shaders that are not fill-bound will see little benefit. In Meta’s own best-case measurement, where 16% of GPU time went to timewarp and was untouched by FFR, the low, medium and high settings improved performance by 6.5%, 11.5% and 21%. Meta also lists lower power use, less heat, longer battery life, and headroom to raise eye-texture resolution at the same cost.
Because the gain depends on whether the app is fill-bound, profile before you turn it on. A fill-bound scene gains from foveation. A scene not bound on GPU fill will see little improvement.
NVIDIA attacked the same waste from the optics side. NVIDIA’s Lens Matched Shading, introduced with the Pascal architecture, renders to a surface that approximates the lens-corrected output. That avoids shading pixels that would be thrown away before reaching the headset display. For a broader look at these techniques in production, see our piece on GPU optimisation for XR workloads. The perceptual side, meaning how much of the peripheral field users actually notice, connects to improving peripheral vision in VR for a wider field of view.
OpenXR 1.1 moved quad-view foveation, along with other widely used extensions, into the core specification. Research keeps pushing the idea further. DeepFovea (November 2019) rebuilt peripheral imagery from sparse samples with a generative network to lower foveation cost, and attention-aware foveated rendering (July 2023) combined gaze with visual-attention models to handle eye-catching content in the periphery.
How much eye-tracking latency can foveated rendering tolerate?
The answer is tens of milliseconds. NVIDIA’s 2017 study found no significant drop in acceptable foveation for total system latencies up to 50–70 ms in VR. Adding 80–150 ms of eye-tracking latency did significantly reduce how much foveation users accepted, but adding 20–40 ms did not.
The setup matters for reading those numbers. The researchers used an HTC Vive with a factory-calibrated SMI eye tracker. The tracker had a minimum latency of about 6 ms and sampled at 250 Hz. End-to-end latency, from eye-image capture to the next displayed frame, was usually 31–35 ms, and about 11 ms longer when a frame dropped. On top of that baseline they added 0, 10, 20, 40, 80 or 150 ms of tracker latency. They tested three foveation methods (subsampling, Gaussian blur and foveated coarse-pixel shading) at fovea sizes of 5°, 10° and 20°. Results for 0, 10, 20 and 40 ms of added latency were statistically indistinguishable, while 80 ms and 150 ms both differed significantly from the low-latency conditions. All three methods suffered at 80 and 150 ms. The authors conclude that, below some threshold, artefact detection is limited only by peripheral perception, and given their measured system latency they place that threshold at an eye-to-image latency of 50 to 70 ms.
Worked comparison: two latency budgets on the same headset
Set the two loops side by side, using the figures above:
- Head pose to photon: the Rift paper puts the ideal below 20 ms, and the 2025 wireless paper sets 14 ms.
- Eye position to foveated image: 50–70 ms tolerated in NVIDIA’s HMD study.
NVIDIA’s paper notes that visual sensitivity drops from about 50 ms before a saccade starts until about 100 ms after it begins. One caution: these results come from one 2017 lab study on one headset and tracker, so treat them as a working budget, not a guarantee for every display and set of content.
Which VR workloads should run on the headset, and which should be offloaded?
Keep unpredictable, latency-critical work on the headset. Heavy work that can be predicted is a candidate for pre-rendering elsewhere. Measurements here come from Furion, a research system that ran high-quality VR on a Pixel phone over 802.11ac Wi-Fi.
Furion’s profiling shows why neither extreme works. Rendering a high-quality VR frame locally on the Pixel took 63–111 ms and pushed the GPU to about 100% utilisation. Streaming a server-rendered frame on demand took about 230 ms. The authors’ key observation is that a frame is made of two parts. Foreground interactions are light but unpredictable. The background environment is heavy because of rich detail and textures, but predictable, since it changes gradually as the user moves. Rendering interactions alone took 12–13 ms on the phone, while the environment alone took 61–104 ms, about 90% of the full-frame time.
So Furion splits the work. The phone’s GPU renders interactions locally. A server pre-renders a panoramic frame for each grid location in the virtual world, and the phone prefetches the frames nearby. Fetching three panoramic frames took 13 ms, less than the 20 ms the user needed to walk to the next grid point. That takes network transfer off the critical path, as long as average bandwidth stays around 130 Mbps. Parallel decoding brought decode time from 45 ms down to 12 ms, which gave a 14 ms end-to-end frame time.
| Rendering approach | Where the work runs | Frame rate on high-quality apps (Furion study) |
|---|---|---|
| Local (mobile) | Everything on the phone GPU | 9–15 FPS |
| Thin client | Everything on the server, streamed as H.264 | 34–37 FPS |
| Cooperative split | Interactions on the phone, environment pre-rendered on the server | 60 FPS |
The split version kept controller latency under 14 ms, rotation under 1 ms and movement under 12 ms. It used up to 37% of the GPU and 65% of the CPU, leaving enough headroom to avoid thermal throttling over long sessions. Bear in mind that Furion targeted a 60 FPS phone display, so its absolute numbers are specific to that setup. The way it divides the work does carry over: put each workload where its predictability and latency tolerance fit.
Power is the other half of the trade. The 2025 wireless VR paper notes that offloading to the cloud can cut on-device compute almost to zero but raises the power spent on wireless communication, within a total budget below 1 W. It also points out that foveated rendering reduces how much data has to be transmitted. Interactions stay local, avoiding network latency. Predictable backgrounds can be pre-rendered and prefetched. Eye-tracked foveation tolerated 50–70 ms of eye-to-image latency in NVIDIA’s study, and foveation reduces what a network link has to carry.
What the history says about the next constraint
Over successive stages, external computers, base stations and controllers gave way to onboard tracking and compute. The arXiv chronology notes that this transfers more responsibility to onboard perception software. Its authors argue that ubiquity now depends on a coherent low-latency perception–action loop, not on portability or component count.
Our reading is that the open engineering question is no longer whether a headset can render a scene. It is which perceptual model sets the budget: where the eye is pointing, what draws attention, and which pixels can be predicted ahead of time. That is the bet the Furion authors made when they argued for software innovation over waiting on hardware. Where that leads for specific uses is covered in future applications of virtual reality.
