Augmented reality: what is it? Augmented reality (AR) is a system that combines real and virtual worlds, interacts in real time and registers virtual content accurately in 3D to real objects. Registration is one of AR’s most basic problems. Registration turns a floating label into an annotation that stays on the bolt it points at. Registration error is also a major failure point for AR systems. Keeping registration consistent means beating latency, the greatest source of registration error.
Augmented reality: what is it, in simple terms?
In simple terms, AR supplements what you already see instead of replacing it. Ronald Azuma put it this way: “Instead of replacing the real world, we supplement it”. IEEE defines AR as the perception of digital information, usually visual, synchronised with objects and places in the physical world around the user. ETSI’s Augmented Reality Framework group calls it the ability to mix spatially registered digital content with the real world in real time.
The US FDA’s definition names the devices. It describes AR as digital imagery overlaid on or mixed with the real world as seen through a camera or display, such as a smartphone, a head-mounted display or a heads-up display, and notes that this is sometimes called mixed or merged reality.
These definitions share a requirement: the digital content has to be registered or synchronised with the physical world. The same model resting on your actual table, and staying there while you walk around it, is AR.
How AR keeps virtual content pinned to the real world
AR runtimes provide sensing, tracking and rendering pipelines. They estimate where the device is (tracking) and the geometry of the surroundings (scene understanding), and draw the virtual content aligned to that viewpoint (rendering).
Tracking: SLAM, inertial sensors, markers and anchors
Common approaches to markerless AR tracking are visual-inertial odometry and SLAM (simultaneous localisation and mapping). Google’s ARCore documentation describes the process. The runtime detects visually distinct feature points in the camera image, uses their movement to compute how the device has moved, and combines that with readings from the inertial measurement unit (IMU) to estimate the camera’s position and orientation over time. Fiducial markers (printed patterns the system knows in advance) are still used when the scene can be prepared for them.
ARML 2.0 calls the link between a virtual object and the physical world an anchor. In the ARML 2.0 model that ETSI summarises, an anchor is either a spatial location tracked with the device’s location and motion sensors, or a visual pattern such as a marker, a QR code or a reference image detected in the camera stream. Shared anchors let several users see the same object at once. ARCore’s Cloud Anchors send an anchor and the feature points around it to the cloud so that other Android or iOS devices in the same space can render content attached to it.
At city scale, Google’s approach does not rely on GPS signals. Google’s Visual Positioning Service locates the device from imagery instead, by matching features in the phone’s camera view against an index built from images with known positions.
Scene understanding: planes, depth and light
Tracking estimates where the camera is. Scene understanding finds the surfaces, depth and lighting the camera sees. ARCore groups feature points that lie on common horizontal or vertical surfaces into geometric planes so that virtual objects can rest on tables and floors. It builds depth maps from the main RGB camera so that virtual objects can collide with real surfaces or disappear behind them. It estimates the average intensity and colour of the scene lighting so that virtual objects can be lit to match. Hit testing projects a ray from a screen tap into the scene and returns the plane or feature point it hits, which is how a user places or selects something.
Liu, Li and Gruteser (MobiCom 2019) pointed out that most AR systems of the time could detect surfaces but not detect and classify complex objects, and that large convolutional networks able to do this are difficult to run on mobile devices at low latency.
Registration is where AR breaks
If real and virtual objects are not properly aligned, the illusion that they coexist collapses. Azuma gives two reasons why this matters more in AR than in VR. The human visual system notices misregistration of just a few pixels, and errors that go unnoticed in a fully virtual environment, such as small head-tracking errors or a slightly misaligned headset, become obvious when the real world sits right behind the graphics.
Of all the sources of registration error in AR, system latency is the largest, outweighing all the others combined. Peter Lincoln’s UNC dissertation states this, citing Holloway (1997) and Brooks (1999). Latency makes virtual content “swim” behind its intended position while the head moves. A discrete frame rate adds judder, where the imagery smears and strobes. Lincoln works out what this means for optical see-through headsets: keeping up with a 10°/s head turn needs less than 25 ms of motion-to-display lag, and a 100°/s turn needs less than 2.5 ms. His approach was to warp the image in the display hardware using the newest tracking data after rendering, which bypasses much of the pipeline’s delay.
How production teams set and defend these budgets, and how they handle tracking loss, is covered in our piece on why AR pilots stall in production.
What is the difference between AR, VR, MR and XR?
AR adds virtual content to a real view. VR replaces the real view completely. Milgram and Kishino (1994) defined mixed reality (MR) through examples of display systems in which real and virtual objects are shown together. They examined three factors that distinguish mixed reality display systems: how much the system knows about the world, how faithfully it reproduces it, and how strongly it creates a sense of presence. XR (extended reality) is the umbrella term for all of these, and the cross-platform OpenXR standard covers both VR and AR devices.
The web standards show the difference in a single field. In WebXR, each session has an environment blending mode: opaque for VR devices, and additive or alpha-blend for AR devices. The term “MR” overlaps with AR in use. The FDA notes that AR is sometimes called mixed or merged reality, and Microsoft describes HoloLens 2 as a holographic computer for collaborating in mixed reality.
For engineering decisions, the operational difference matters more than the definitional one. AR and VR differ in hardware envelope, content authoring cost, session length, input methods and failure modes. Registration marks a key difference. A VR world only has to agree with itself, while an AR overlay has to agree with the room. The full side-by-side comparison is in our guide on how to tell augmented reality and virtual reality apart.
What hardware do you need to deploy AR in 2026?
You need a display that combines the two views, sensors that track the device and the scene, and compute to process them in real time. AR hardware pairs a display with sensors such as cameras and IMUs to register virtual content to the environment. Bimber and Raskar (MERL, 2006) group AR displays into three families: worn on the head, held in the hand, or placed in the environment and detached from the user.
AR display classes compared
| Display class | How real and virtual are combined | Strengths | Limits |
|---|---|---|---|
| Optical see-through HMD | Optical combiners (half-silvered mirrors or transparent displays) superimpose graphics on the directly viewed world | Essentially unhindered, instant view of the real world; if it fails, the wearer still sees the room | Overlay field of view typically 20° to 60°; hard to make virtual objects block real ones |
| Video see-through HMD | Head-mounted cameras capture the world and graphics are mixed into the video electronically | Handles occlusion and blending well; can guarantee real-virtual registration | Camera viewpoint differs from the eye’s; if it fails, the wearer can lose the real-world view entirely |
| Handheld (phone, tablet) | Video see-through on a screen held at arm’s length | Processor, memory, display and input in one widely owned device | Image analysis and rendering are processor- and memory-heavy, which risks delay and low frame rates on weaker devices |
| Spatial or projection | Projectors draw directly on physical surfaces, or fixed combiners sit in the environment | Better ergonomics, a theoretically unlimited field of view, scalable resolution | Optical distortion changes with the viewer’s moving, off-axis viewpoint and needs dynamic correction |
The optical versus video choice also involves trust and failure-mode trade-offs. Rolland, Holloway and Fuchs, comparing the two, observed that video see-through may meet resistance in the workplace because it removes the direct view of the world, a problem that grows in safety-critical work. They concluded that no single type of see-through system is right for every application. The failure mode deserves first place in that choice: a technician next to live equipment should still see the room if the headset fails.
What current devices look like
Current headsets put a lot of sensing and compute into one unit. Apple’s specification for Vision Pro lists an M5 chip with a 10-core GPU, a separate R1 chip rated at 12 ms photon-to-photon latency, six world-facing tracking cameras, a LiDAR scanner and four IMUs, with a weight of 750 to 800 grams and up to 2.5 hours of general use on battery. Those figures come from Apple’s own specification. Microsoft documents HoloLens 2 as an untethered device built on a Qualcomm Snapdragon 850 with a second-generation custom holographic processing unit, providing 6DoF tracking and real-time environment meshing, with 2 to 3 hours of active use.
On Android phones, hardware support depends on ARCore certification. Google certifies a device for ARCore only after checking its camera, motion sensors, design and CPU performance, because motion tracking combines camera images with motion sensor input and has to run in real time. According to Google, over 88% of active ARCore devices supported the Depth API as of May 2026.
Why AR needs GPU compute, and where that processing runs
AR needs serious compute because image analysis and rendering are processor- and memory-intensive and must run in real time. At 60 fps that leaves 1000 ms ÷ 60 ≈ 16.7 ms per frame, which is the inter-frame time at 60 fps that Liu, Li and Gruteser’s system worked within. Rendering is only part of the load. Rendering corrections run on graphics hardware too: Bimber and Raskar describe implementing optical refraction correction as a vertex shader on the GPU. OpenXR abstracts that rendering pipeline (viewports, stereoscopy, the compositor) across devices, and Vulkan offers a low-overhead graphics and compute API for real-time work. The cost of moving existing compute-shader code onto Vulkan is a separate question, covered in our article on porting compute-shader workloads to Vulkan compute.
Object detection is the part of scene understanding that overloads a device. Liu, Li and Gruteser (2019) reported that lightweight object detection models took more than 500 ms on high-end smartphones with GPU support, and that even a Tegra TX2 mobile GPU needed more than 50 ms to run detection on an HD frame. Power is also a constraint. A 2024 paper on smartphone AR notes that camera-based tracking can degrade performance through high power use, causing overheating and shorter battery life, while IMU-based tracking is less resource-intensive.
The work can run on the device, at the edge, or on a central server or cloud renderer.
Where AR processing runs: device, edge or central GPU
| Placement | What belongs there | Evidence and limits |
|---|---|---|
| On the device | Tracking, final rendering, late-stage warping: anything that must react within the frame | Liu et al. (2019) ran motion-vector tracking in 2.24 ms on a Jetson TX2, leaving the rest of the 16.7 ms frame for rendering |
| Edge GPU | Heavy perception such as object detection, offloaded | Offloading to an edge GPU raised detection accuracy by 20.2% to 34.8% at 60 fps, and, without rendering, the offloading work used only 15% of the device’s CPU and 13% of its GPU |
| Central GPU server or cloud | High-responsiveness inference for a few users; streamed photorealistic rendering for AR devices | One study found that only a powerful central server met high-responsiveness targets, and only for a few users; distributed edge devices served many more users at 100 ms and 500 ms responsiveness tiers |
On streaming, NVIDIA describes CloudXR 6.0 as an OpenXR bridge that renders and encodes on RTX servers and streams the result to AR, VR and spatial computing devices. This decouples heavy rendering from the headset, so the network sits between rendering and display. Our reading of the measurements is that the useful split keeps anything tied to head motion on the device and sends anything that can tolerate a frame or two of delay outward. Liu et al. designed for that boundary by separating the rendering pipeline from the offloading pipeline and filling the gap with on-device tracking.
Where is augmented reality actually used in 2026?
TechnoLynx’s assessment is that the industries where AR has credible production economics in 2026 share three properties: the work is hands-busy or eyes-on-equipment, errors or returns are expensive, and the overlay content can be authored once and reused many times. If any of the three is missing, the business case gets weak.
Manufacturing, maintenance and field service
Caudell and Mizell (1992) at Boeing used optical see-through HMDs with head tracking to stabilise computer-generated diagrams on specific locations of a real assembly board, and they named sufficient registration as the main difficulty. BMW and Volkswagen, along with Boeing, have also been reported to use AR on assembly lines. ETSI’s framework lists the maintenance tasks in detail: presenting work-order information hands-free on a head-up display, guiding each step of a complex procedure in the right order over the real equipment, rehearsing a sensitive operation in AR before touching critical equipment, and annotating a situation to share or record it. AR could also guide picking and storeroom work, and we cover the freight side in augmented reality in cargo management.
Surgery and medical imaging
The FDA gives, as an example, an AR system that overlays medical images onto a patient during an operation to guide the surgeon. It lists surgery planning and intraoperative procedures among the treatment domains, and distraction in the operating room among the risks. The research history goes back further. A UNC group overlaid a 3D ultrasound model of a fetus onto the mother’s abdomen, and a cranio-facial planning system aimed to superimpose CT skull data onto the patient.
Flight and military training
Red 6’s ATARS system brings AR into live flight training for pilots. The system projects virtual adversaries into the pilot’s view during live flight, so crews experience real G-forces and workload while fighting threats that need no extra aircraft. In simulators, University of Illinois researchers found that showing an AR flight path only when a student drifted off it trained landings better than showing it all the time. Training design matters as much as the overlay itself.
Navigation
On Pixel phones, Google Maps overlays walking directions on the camera view, built on ARCore. In-vehicle heads-up displays overlay route guidance for drivers, and our article on augmented reality in the automotive industry goes further into that case. Research finds AR navigation can improve wayfinding and driver situational awareness, but distraction, cognitive load and occlusion all matter when the use is safety-critical.
Retail and consumer try-on
Furniture retailers including IKEA, Houzz and Wayfair offer apps that let shoppers see products in their homes before buying. IKEA’s Place app, announced in 2017, placed true-to-scale 3D models of over 2,000 products in a room through the phone camera. Phone cameras also let shoppers try on glasses, and we cover makeup try-on in AR makeup try-on in beauty.
Some AR deployments fail in pilot before they reach production. The hardware, latency and content constraints behind that are covered in our article on why AR pilots stall in production.
Registration tolerance is the first requirement to set
Registration tolerance, not the display, is the first thing to specify. Once you know how far the overlay is allowed to drift and how fast the user’s head moves, most of the other decisions follow: optical or video see-through, how much perception runs on the device, and whether an edge GPU has to sit in the same building. For more on how the computer vision side of those overlays is built, see creating AR experiences with computer vision.
