“AI video composer” sounds like one tool that ingests raw feeds and generated layers and hands you a finished frame. In post-production, that reading is roughly right. On air, it is the reason overlays drift off the action and occlusion goes wrong at the worst possible moment. The confusion is worth naming precisely because the words are the same. An offline composer and a live compositing stage share a vocabulary — layers, mattes, keying, occlusion — but they answer to opposite masters. The offline composer optimizes for the best possible frame with no hard deadline. The live composer optimizes for a predictable frame that arrives on time, every time, because the next play does not wait for your model to finish thinking. What does an AI video composer actually do in a broadcast pipeline? Strip away the marketing and an AI video composer in a live context is a deterministic compositing stage. It sits between the perception layer that understands the scene and the output that goes to air. Its job is narrow and unforgiving: for each incoming frame, take the frame-locked inputs it is handed — the camera feed, the tracked pose or object positions, the graphic layers, any generated mattes — and decide how those layers stack, where they occlude one another, and what the final pixels are. Then it emits, within a fixed budget, before the frame is due. The “AI” part is real but bounded. Segmentation models produce mattes that separate a player from the pitch. Pose estimation places a graphic relative to a moving body. Detection and tracking keep a lower-third anchored to the right object as it moves through frame. But the composer itself is not where you want an unbounded model making open-ended decisions. It is where those model outputs get assembled into a frame under a clock. That distinction — perception can be probabilistic, but compositing must be deterministic — is the whole game. When keypoint estimation feeds an overlay, the same principles that govern driver-facing AR pose estimation apply here: the pose is an input to compositing, not a substitute for it. We treat the composer as the stage that must never surprise you, fed by models that occasionally will. Where the naive read breaks: offline quality vs. on-air timing The failure class is timing drift. An offline composer is free to spend 200 milliseconds on a frame if the result is better, because it renders faster than real time and buffers ahead. A live composer has no such freedom. At 60fps the frame budget is one frame period — sub-16.7ms — and the compositing stage is only one consumer of that budget alongside ingest, perception, and output encoding. Blow the budget once and you either drop the frame or ship it late, and on live sports either outcome is visible. Here is the trap: a composer built on the offline mental model looks flawless in a demo reel. The demo is pre-recorded, the model runs to completion on every frame, and the compositing is beautiful. Then it goes live, the load spikes when three players cluster and the segmentation model’s per-frame cost jumps, and the composer that averaged 12ms in the demo now spikes to 25ms exactly when the action is busiest. The overlay lags the action by a frame or two, the occlusion mask is computed against last frame’s positions, and the graphic sits on top of a player it should sit behind. None of that shows up until the moment that matters, which is why we insist on separating the two concerns before a line of compositing code is written. A live composer is judged by its worst frame under load, not its average frame in a demo — that is the reframe most teams arrive at only after their first bad broadcast. Offline composer vs. live compositing stage: a decision table The two share a name and diverge on almost everything that determines whether they work on air. This table is the fastest way to check which one a given design is actually building. Dimension Offline AI composer Live compositing stage Latency target Best quality, unbounded Bounded per frame (sub-16.7ms @ 60fps) Failure mode None visible (re-render) Dropped/late frame, overlay drift Model execution Runs to completion Must fit the per-frame budget or degrade gracefully Input timing Batched, out of order OK Frame-locked, synchronized to cadence Occlusion decision Against final scene state Against the current frame’s tracked state Determinism Optional Mandatory — same inputs, same timing Buffering Renders ahead freely Minimal; buffering is added latency Judged by Average frame quality Worst frame under peak load The right-hand column is the spec for anything that goes to air. If a design borrows the left-hand column’s assumptions — unbounded model time, buffering ahead, quality-first — it is an offline tool wearing a live costume. How does a live composer stay inside a single-frame budget? The budget is not a suggestion; it is the contract with the broadcast clock. Staying inside it comes down to three engineering choices, each of which trades a little theoretical quality for guaranteed timing. First, frame-locked ingestion. Every input the composer consumes — the camera frame, the tracked positions, the mattes — must arrive tagged to the same frame and synchronized to cadence. If pose data arrives one frame late, the composer must know that and decide whether to use last frame’s positions or wait, rather than silently compositing mismatched inputs. Synchronization is upstream of compositing, and getting the feeds frame-locked in the first place is partly a physical-layer problem — the low-latency, high-bandwidth links described in our note on active optical cables for live broadcast AR exist precisely so the composer is not starved waiting for data. Second, bounded per-frame cost on the GPU. The compositing passes — keying, matte compositing, occlusion resolution, graphic blending — should be structured so their cost is predictable rather than data-dependent in ways that spike under load. Fusing passes so the GPU does not round-trip through memory between each stage is one lever; our explainer on fusing GPU passes for frame-locked AR overlays walks through why a mega-kernel approach keeps the per-frame cost flat. Keeping layer data resident and avoiding host round-trips matters too, which is where unified virtual memory behaviour for GPU inference either helps or quietly costs you a page fault at the wrong time. Third, graceful degradation instead of hard failure. When the budget is at risk — three players clustered, segmentation cost spiking — the composer should shed quality in a controlled way (coarser matte, simpler occlusion resolution) rather than blow the frame. A composer that always ships something on time, occasionally slightly rougher, beats one that ships perfect frames until it drops one on the winning goal. This is validated cheaply before deployment: profiling the perception models the way you would YOLO inference on GPU under realistic batching tells you their worst-case per-frame cost, which is the number the budget has to survive. What inputs does a live composer need to lock overlays to the action? An overlay locked to the action is only as good as the inputs feeding the lock. In practice a live AI composer needs four classes of input, each frame-synchronized: The camera feed itself, frame-locked and time-stamped, as the base layer and the reference for everything else. Frame-locked pose or tracked object positions — where the player, ball, or car is this frame, not last frame. This is what anchors a graphic to a moving target. Layer feeds — the graphics, lower-thirds, virtual advertising, and generated elements that will be composited in, each with its own z-order intent. Mattes and segmentation masks — the per-pixel information that says what is foreground and what is background, so the composer can put a graphic behind a player instead of over their face. Miss any one of these, or let it arrive out of sync, and the composer either guesses or drifts. The occlusion decision in particular depends entirely on the matte and the tracked positions being for the same frame. Composite a current-frame graphic against a last-frame matte and you get exactly the wrong-occlusion artifact that reads as broken on a broadcast. Where does automated compositing hold up live, and where does it still need an operator? Automated compositing has become genuinely reliable for the repeatable, well-defined cases: anchoring a persistent graphic to a tracked object, keying a known background, resolving occlusion when the scene geometry is clean and the segmentation is confident. In configurations we have worked with, these run inside the frame budget consistently because their per-frame cost is bounded and their decisions are structural, not judgemental (observed across broadcast-compositing engagements; not a published benchmark). Where it still needs a human is judgement under ambiguity: a matte that fails when two players overlap and the segmentation confidence collapses, a graphic that is technically correct but editorially wrong for the moment, a scene the models were never trained for. The mature live setup does not try to remove the operator — it frees the operator from the mechanical work so their attention lands on the frames the composer flags as low-confidence. Determinism helps here too: because the composer behaves the same way given the same inputs, an operator can predict and pre-empt its behaviour rather than fighting a system that surprises them. FAQ What does working with an AI video composer involve in practice? In a live broadcast pipeline it is a deterministic compositing stage sitting between perception and output. For each frame it takes frame-locked inputs — camera feed, tracked positions, graphic layers, mattes — decides layering and occlusion, and emits within a fixed per-frame budget. The AI models feed it (segmentation, pose, tracking) but the composing itself must be predictable, not open-ended. What is the difference between an offline AI video composer and a real-time compositing stage for live broadcast? An offline composer optimizes for the best possible frame with no hard deadline and can re-render freely. A live compositing stage must be bounded and predictable per frame, because the frame is due at broadcast cadence and there is no re-render. The divergence point is timing: unbounded quality versus bounded, on-time output, judged by the worst frame under load rather than the average. How does an AI composer stay within a single-frame budget at broadcast cadence? Through frame-locked ingestion (all inputs synchronized to the same frame), bounded per-frame GPU cost (fused passes and resident data so cost stays flat under load), and graceful degradation (shedding quality in a controlled way rather than dropping a frame). At 60fps that budget is sub-16.7ms, shared with ingest, perception, and encoding. What inputs does a live AI composer need to lock overlays to the action? Four frame-synchronized classes: the frame-locked camera feed, frame-locked pose or tracked object positions, the graphic layer feeds with their z-order intent, and mattes or segmentation masks for foreground/background separation. The occlusion decision in particular requires the matte and tracked positions to be for the same frame, or overlays occlude wrongly. Where does automated AI compositing hold up in live production, and where does it still need a human operator? It holds up for repeatable, well-defined cases — anchoring graphics to tracked objects, keying known backgrounds, resolving occlusion with confident segmentation — because their per-frame cost is bounded. It still needs a human for judgement under ambiguity: collapsed segmentation confidence, editorially wrong choices, or untrained scenes. The mature setup frees the operator to focus on the frames the composer flags as low-confidence. How does deterministic compositing prevent overlay drift and wrong occlusion during a live event? Because the composer produces the same output and timing for the same inputs, it never silently spends more than its budget or composites mismatched frames. Frame-locked inputs mean the occlusion mask is computed against the current frame’s positions, not last frame’s, which is what prevents a graphic sitting on top of a player it should sit behind. Determinism turns the composer into something an operator can predict rather than fight. A useful way to close is to notice what the budget really constrains. The composer’s per-frame compositing cost has to fit inside the deterministic window the broadcast cadence allows — and whether it does is not something you find out on air. It is something you validate ahead of time by measuring the compositing stage against the frame budget, the kind of check an A1 GPU Audit is built to run, and by treating the whole broadcast overlay path as a timing contract rather than a quality-only pipeline, the way our work on live broadcast media systems frames it. The failure class to remember is timing drift: a composer that looks great in the reel and misses the moment on air.
