AR beauty try-on is the highest-volume consumer AR surface in production today, and it is also the one where the most experiences die before they render. The reason is rarely the makeup model. It is the cold-start path on a tapped link, on a device the team did not test, on a network that is not the office Wi-Fi. In our experience working with brands deploying AR features through web and social channels, the engineering question is not “how realistic is the lipstick” but “did the user still have the page open when the first frame arrived?” This article walks through how the AR beauty stack actually behaves in the wild — the CV pipeline that runs behind virtual makeup, the asset and model loading order that determines whether the user sees a brand or a spinner, and the device-fragmentation traps that turn a polished demo into an abandoned tap. For the parent argument about AR advertising surfaces as a whole — 3D billboards, social filters, and native ad units — see AR Advertising and 3D Billboards: The Cold-Start Problem. What runs behind a virtual makeup try-on? The user-visible experience is “point camera at face, see lipstick.” The pipeline behind it is a chain of CV components that must each meet a frame budget. A typical web-AR or social-AR beauty stack runs, per camera frame: Face detection — a small detector (often a MobileNet- or BlazeFace-style network) locates the face bounding box. Cost is low and roughly constant. Landmark regression — 68 to 468 facial landmarks are regressed inside the detected box. MediaPipe Face Mesh is the de facto reference here; vendor stacks like Banuba, DeepAR, and Snap’s Camera Kit ship their own equivalents. 3D head pose and expression — landmarks are lifted into a parametric face model (a 3DMM variant) so that virtual makeup tracks pitch, yaw, and lip motion. Segmentation for the makeup region — lips, eyelids, cheeks, and hair each need their own segmentation mask. These are usually small U-Net-style networks running per region. Render and composite — the makeup texture is blended on the segmented region with lighting compensation, then composited back into the camera feed. The operationally relevant measure here is sustained per-frame latency under realistic load, not peak throughput on a flagship phone. observed-pattern: across the AR beauty deployments we have audited, the segmentation and rendering steps — not the landmark network — are the most common source of frame-time variance, especially when lighting compensation is done on the CPU instead of in a fragment shader. That has a direct consequence for how the pipeline should be built. The CV components must be compiled or quantised down to the cheapest representation that still preserves landmark stability across head rotation. Stable landmarks matter more than precise ones; jitter is what users perceive as “the makeup is sliding off my face.” The CV pipeline at a glance Stage Typical model Frame budget (mid-tier mobile) Common failure Face detect BlazeFace / MobileNet-SSD 3–5 ms Lost on profile angles Landmark regression MediaPipe Face Mesh, 468-point 8–12 ms Jitter under low light Segmentation (lips/eyes) Small U-Net per region 6–10 ms Bleeding outside lip line Render and composite Fragment shader 4–8 ms CPU fallback on older GPUs Total per frame — ~25–35 ms Hitting 30 fps is the floor Numbers above are observed ranges from production AR beauty stacks we have looked at; they are planning heuristics, not a benchmarked rate for any specific vendor. Why cold-start is the real KPI A user who taps an AR ad has, in practice, three to five seconds of patience. The clock starts on tap, not on first frame. Everything that happens before the camera viewfinder shows the first composited frame is dead time competing with the user’s decision to leave. A cold-start path on a typical web-AR beauty experience looks roughly like: HTML and bootstrap JS download — 200–400 ms on LTE. WASM runtime for the CV stack download and instantiate — 500–1500 ms. Model weights (landmark + segmentation) download — 1–3 MB compressed, another 300–800 ms. Camera permission prompt — variable, but blocks everything behind it. First inference warm-up — 100–300 ms on the first frame because shader compilation and tensor allocation happen lazily. Add those and a cold-start budget of three seconds is tight, and a five-second budget is realistic only when the asset pipeline has been engineered for it. observed-pattern: in audits across consumer AR placements, the teams that miss the cold-start budget almost always authored the experience like a film — assets bundled in narrative order — instead of streaming the camera-critical components first. The fix is structural. Ship the landmark model and the camera bootstrap in the first network round-trip. Defer the segmentation models and the makeup textures until after the first viewfinder frame paints. Compile shaders ahead of time when possible. Treat the model weights as critical render assets, not as data. How AR beauty integrates into the e-commerce funnel The brand-side question is not “does the try-on look good” but “does it move the funnel.” There are two integration patterns that show up repeatedly. The first is try-on as a product-page widget. The AR experience replaces the static product photo on a PDP (product detail page). Engagement is measured as the share of PDP visits that activate the camera. observed-pattern: when the try-on is gated behind a download, activation collapses; when it runs in-browser, activation is meaningfully higher, though the exact lift depends on traffic mix and is not portable between brands. The second is try-on as a social-ad surface. The experience lives inside Instagram, TikTok, or Snap, authored against that platform’s AR SDK (Spark AR before its sunset, Meta’s successors, TikTok Effect Studio, Snap Lens Studio). Here the funnel is the platform’s: engagement, share, swipe-up to PDP. The CV pipeline is largely the platform’s; what the brand controls is the makeup model, the asset weight, and — critically — the fallback for devices the platform deems too weak. Both integration patterns share the same operational truth: the brand impression only lands if the experience renders. Conversion attribution and creative quality are downstream of that. Device fragmentation and the fallback question Consumer AR is a long-tail device problem. A web-AR beauty experience will be served to a four-year-old mid-range Android alongside a current-generation iPhone. Their GPU capabilities differ by an order of magnitude, and their browsers expose different subsets of WebGL and WebGPU. Three practical fallback decisions matter: What renders when the camera is denied or unavailable? A static product image is a coherent fallback; a broken viewfinder is not. What renders when the device cannot sustain 30 fps? Some stacks drop the segmentation step and fall back to a simpler bounding-box overlay. Some drop to a still photo with a swatch slider. Both are acceptable; a stuttering try-on is not. What renders when WebGL2 or WebGPU is unavailable? The CV stack needs a CPU path or a graceful degradation surface. Shipping a JavaScript fallback for tfjs is non-trivial and rarely worth it for the beauty use case; a static experience is usually the right answer. The teams that ship robust AR beauty experiences treat the fallback path as a first-class engineering deliverable, not as an error state. The teams that treat it as an afterthought see their reported “AR engagement” numbers quietly inflated by users who never actually saw a composited frame because the renderer silently fell through to the camera passthrough. Where this is heading Two shifts are visible in the AR beauty stack right now. The first is generative try-on: instead of compositing a pre-authored makeup texture, a diffusion-style model generates the made-up face directly. This collapses several pipeline stages but moves the latency problem to model size, which is currently incompatible with cold-start budgets on consumer devices. It will be a server-side experience first. The second is personalised recommendation tied to the try-on session. The same landmark and segmentation outputs that drive the makeup overlay can feed a skin-condition assessment — dryness, oiliness, undertone — and recommend products against the brand catalog. The engineering risk is conflating CV output with medical claims; the framing has to stay decision-support, not diagnosis. Both directions reinforce the same point this article opened with. The brand that wins the AR beauty surface is the one whose engineering team treats the pipeline like infrastructure with a latency SLO, not like a creative deliverable. The CV stack, the asset loader, and the fallback path decide whether the try-on becomes a measurable contributor to the funnel or another novelty engagement metric that does not survive contact with attribution. For the broader argument about AR advertising as a category — 3D billboards, native ad units, and social filters under the same cold-start discipline — the parent piece is AR Advertising and 3D Billboards. A common failure class here is the AR ad that hits its target render quality on the developer’s flagship device and silently degrades on the long tail; A1 GPU Audit with cold-start instrumentation is the artifact that catches it before launch. FAQ What are the production patterns for AR advertising — billboards, social filters, native ads? The three surfaces share a CV pipeline shape (detect, track, segment, render) but differ on the cold-start path. 3D billboards run on controlled hardware with no cold-start. Social filters inherit the platform’s runtime and cold-start. Native and web-AR ads carry the full cold-start cost — bootstrap, WASM, model weights, shader warm-up — and live or die on it. How does AR beauty try-on integrate measurably into a brand’s e-commerce funnel? Two integration patterns dominate: try-on as a product-page widget (measured by camera-activation rate on PDP visits) and try-on as a social-ad surface (measured by the platform’s engagement and click-through metrics). Both require the experience to actually render — attribution downstream of a broken viewfinder is noise. Which AR advertising examples actually drive ROI versus novelty engagement? The ones whose cold-start path was engineered, whose fallback for weak devices is a coherent static experience rather than a broken viewfinder, and whose engagement metric counts composited frames rather than camera-permission grants. Novelty engagement inflates when fallback paths are silent. What CV pipeline runs behind virtual makeup, hair, and skincare try-on at scale? A chain of face detection, 468-point landmark regression, 3D head-pose lifting, per-region segmentation (lips, eyes, cheeks, hair), and shader-based rendering. MediaPipe Face Mesh is the de facto landmark reference; vendor stacks ship their own equivalents. The frame budget on mid-tier mobile is roughly 25–35 ms; segmentation and rendering are the typical variance sources. How do AR newspaper and billboard ads handle device fragmentation and cold-start UX? Billboards sidestep fragmentation by running on controlled hardware. Newspaper and web-AR placements cannot; they ship a tiered fallback (full try-on, simplified overlay, static experience) selected by device capability detection, and they engineer the asset loading order so the camera-critical components arrive before the experience-critical ones. Where are AR beauty and advertising applications evolving — generative try-on, personalization, social integration? Generative try-on (diffusion-based, server-side first) is collapsing pipeline stages at the cost of latency. Personalised recommendation reuses the landmark and segmentation outputs to drive product suggestions. Social integration is converging on platform-native AR SDKs as the distribution surface, with brands authoring against the platform rather than against the open web.
