A common assumption trips up teams evaluating object tracking software: that once a detector finds the objects, following them from frame to frame is essentially free. It is not free, and it is not part of the detection budget. Tracking is a separate pipeline stage with its own compute profile — and that profile is usually shaped by association logic and motion models that run on the CPU, not by the GPU cycles the detector consumes. That distinction is the whole point of this article. When teams fold tracking into the detection budget and size both together, they either over-provision GPU capacity for work the CPU handles cheaply, or they starve the detector because the tracking overhead was invisible until it wasn’t. A stage-aware view lets you place tracking where its economics actually work. How does Object Tracking Software actually work? Object tracking is the task of maintaining a consistent identity for each object across a sequence of frames. A detector run on a single frame tells you there is a car here, a person there — but it has no memory. Run the same detector on the next frame and you get another unordered list of boxes. Tracking is the layer that says the car in frame 200 is the same car that was in frame 199, assigning and preserving a stable ID. In practice, that reduces to two problems solved every frame. First, prediction: given where an object was and how it was moving, estimate where it should appear now. Second, association: match the current frame’s detections to the existing tracked objects, decide which detections are new objects, and decide which tracks have disappeared. The classic implementation of prediction is a Kalman filter over each object’s position and velocity; the classic implementation of association is the Hungarian algorithm run over a cost matrix of predicted-versus-observed positions. SORT and its descendants (DeepSORT, ByteTrack) are built on exactly this skeleton. None of that is inherently a matrix-multiply workload. A Kalman filter updates a handful of small state vectors. The Hungarian algorithm solves an assignment problem over a matrix whose size is number of tracks × number of detections — tens by tens in most scenes, not the millions of parameters a detection network touches. This is bookkeeping, not tensor math, and bookkeeping is what CPUs are good at. How the Tracking Stage Differs From the Detection Stage That Feeds It Detection and tracking sit next to each other in the pipeline and are easy to conflate, but they diverge on every axis that matters for hardware sizing. Detection is a dense, data-parallel forward pass through a convolutional or transformer backbone — YOLO, RT-DETR, or similar — decoding a frame into feature maps and then into boxes. That is precisely the workload GPUs exist for, and it is where the decode-and-inference cost of a video-analytics pipeline concentrates. We cover that side in object detection in videos and what it costs on the decode path, which is the natural companion to this piece. Tracking, by contrast, is a sequence of small, latency-sensitive, control-heavy operations. The association step depends on the result of the current frame’s detection, so it cannot start until detection finishes — it is serially downstream. And because it operates on the compact output of detection (a list of boxes, not a full image tensor), the data it moves is tiny. Feeding that tiny workload to a GPU usually means paying host-to-device transfer latency for an operation the CPU would have finished before the copy completed. The citable distinction: detection is a GPU-appropriate, throughput-bound forward pass, while the core tracking loop — Kalman prediction plus Hungarian association — is a CPU-appropriate, control-bound sequence. Sizing them as one budget hides this and leads to placement mistakes. (observed pattern across TechnoLynx video-analytics engagements; not a published benchmark.) Is Object Tracking CPU-Bound or GPU-Bound? For the association-and-motion core, it is typically CPU-bound. What drives that is the shape of the computation, not a preference: small state updates and an assignment problem over a small matrix do not saturate thousands of GPU cores, and the per-frame data volume is too small to amortise the transfer cost of getting onto the accelerator in the first place. There is an important exception, and it is worth naming precisely because it is where the CPU-versus-GPU line actually falls. Detection-Based Tracking vs Appearance / Re-Identification Tracking The compute profile of tracking depends heavily on how association is done. Detection-based (motion-only) tracking — SORT, ByteTrack — associates purely on geometry: predicted position versus observed position, plus detection confidence. There is no neural network in the tracking stage itself. This is the archetypal CPU-bound tracker. Appearance / re-identification (re-ID) tracking — DeepSORT and its successors — adds a learned appearance embedding to each detection so objects can be re-matched after occlusion or after leaving and re-entering the frame. Computing that embedding is a neural-network forward pass, typically a small CNN, and that sub-step is GPU-appropriate. But note what is actually on the GPU: the embedding extractor, not the tracking logic. The Kalman filter and the assignment step stay on the CPU regardless. The table below summarises where each piece lands. Object Tracking Stage: Compute Placement Sub-component What it does Typical placement Why Motion prediction (Kalman) Predicts next-frame position from velocity CPU Tiny state updates; no parallelism to exploit Association (Hungarian) Matches detections to existing tracks CPU Small assignment problem; control-heavy, latency-sensitive Track lifecycle Creates, ages out, and confirms tracks CPU Pure bookkeeping Appearance embedding (re-ID only) Learned feature vector per detection GPU (if present) A CNN forward pass — data-parallel, benefits from acceleration The practical reading: a motion-only tracker adds essentially zero GPU load. A re-ID tracker adds a bounded, additional GPU forward pass per detection — which you size separately, not by lumping it into detection. How Isolating the Tracking Stage Changes a Cost-per-Analytics-Hour Estimate This is where the distinction pays for itself. Consider a worked example with explicit assumptions. Suppose you are analysing 20 camera streams. Detection at your target frame rate needs, say, a GPU-hour’s worth of accelerator time per analytics-hour — that number comes from your detector, resolution, and frame rate, and you would measure it, not guess it. The naive plan sizes tracking into that same GPU budget “to be safe,” so you provision extra GPU headroom for a stage you never profiled. Now isolate the stage. A motion-only tracker for 20 streams runs comfortably on a few CPU cores you are already paying for as part of the host that decodes the video. The GPU line for tracking is zero. Your cost-per-analytics-hour for tracking collapses to the marginal cost of CPU threads, not the marginal cost of GPU capacity — and GPU capacity, per NVIDIA’s published data-center pricing, is the expensive line in any video-analytics bill of materials. The measurable outcome is a defensible placement decision: tracking stays on the CPU, GPU capacity stays reserved for detection and classification, and your utilisation numbers stop lying to you. If you had lumped the stages together you would have carried idle GPU capacity — the underutilisation pattern that quietly inflates cost-per-analytics-hour. That is the same stage-economics reasoning we walk through for inference pipelines in what GPU/CPU stage economics mean on H100. Profiling, not vendor positioning, is what decides where GPU acceleration earns its cost. A per-stage profile of a real computer vision pipeline will show the tracking line as a thin CPU sliver next to a thick detection GPU bar — and being able to read that line is exactly the vocabulary this article is meant to give you. Where Does Object Tracking Sit Between Server-Side and Edge Deployment? The placement choice is driven by latency tolerance and by how many streams share one machine. Server-side, you are usually consolidating many streams onto a few GPU-equipped hosts. Detection batches across streams to keep the GPU fed; tracking runs per-stream on the host CPUs. This is the common broadcast and media-telecom analytics shape, where dozens of feeds converge on a rack and the economics reward keeping the GPU busy only with the work that needs it. At the edge — a camera-adjacent box, or something like a small accelerator paired with an ARM host — the calculus shifts. The host CPU is weaker, so even the modest association workload competes with everything else, and a re-ID embedding may need a small edge accelerator. The handoff between the CPU host and the accelerator becomes the thing you profile, a pattern we unpack for constrained hardware in how edge accelerators handle the CPU-accelerator handoff. The tracking stage does not change what it is between these deployments; what changes is how much CPU headroom you have to spend on it. FAQ How should you think about object tracking software in practice? Tracking maintains a stable identity for each object across frames, sitting on top of a detector that has no frame-to-frame memory. In practice it solves two problems every frame: predicting where each tracked object should move (typically a Kalman filter) and associating current detections to existing tracks (typically the Hungarian algorithm). SORT, ByteTrack, and DeepSORT are all built on this prediction-plus-association skeleton. How does the tracking stage differ from the detection stage that feeds it, and why is that distinction important for hardware sizing? Detection is a dense, throughput-bound forward pass through a neural backbone — exactly what GPUs are built for. Tracking is a sequence of small, control-heavy, latency-sensitive operations that run downstream of detection on its compact box output. Sizing them as one budget hides that difference and leads teams to provision GPU capacity for work the CPU handles cheaply. What is the compute profile of object tracking — is it typically CPU-bound or GPU-bound, and what drives that? The core motion-and-association loop is typically CPU-bound. That is driven by the shape of the work: tiny state updates and a small assignment problem do not saturate GPU cores, and the per-frame data is too small to justify the host-to-device transfer cost. Only a re-identification appearance embedding — a neural sub-step — is GPU-appropriate. What is the difference between detection-based tracking and appearance/re-identification tracking, and how does that change the compute profile? Detection-based tracking (SORT, ByteTrack) associates purely on geometry with no network in the tracking stage, so it stays entirely on the CPU. Re-ID tracking (DeepSORT) adds a learned appearance embedding per detection, which is a small CNN forward pass that belongs on the GPU. Even then, only the embedding moves to the GPU; the Kalman filter and assignment step stay on the CPU. How does isolating the tracking stage change a cost-per-analytics-hour estimate compared with lumping it in with detection? Isolating tracking reveals that a motion-only tracker adds essentially zero GPU load, so its cost collapses to CPU threads you are already paying for. Lumping it into detection forces you to carry idle GPU headroom, inflating cost-per-analytics-hour. The result of isolating it is a defensible placement decision with honest utilisation numbers. Where does object tracking sit between server-side and edge deployment, and what drives that choice? Server-side, many streams consolidate onto GPU hosts: detection batches on the GPU while tracking runs per-stream on host CPUs. At the edge, a weaker CPU means the association workload competes for scarce headroom and a re-ID embedding may need a small edge accelerator. Latency tolerance and streams-per-machine drive the choice; the tracking stage itself does not change. When does object tracking justify GPU acceleration and when should it stay on CPU? Motion-only tracking should stay on the CPU — it has no workload the GPU can accelerate meaningfully. GPU acceleration is justified only for the appearance-embedding sub-step of re-ID tracking, and even then you size that forward pass separately rather than moving the whole tracking loop. The deciding evidence is a per-stage profile, not a vendor’s positioning. What This Leaves You to Measure The vocabulary here — prediction, association, re-ID embedding — is enough to read a profiling result, not to skip one. Before you commit a placement, the open question for your own pipeline is where the tracking line actually falls once your detector, resolution, frame count, and re-ID choice are fixed: is it the thin CPU sliver the geometry predicts, or has an oversized re-ID embedding quietly pulled a second GPU forward pass into the budget? That is the tracking-stage line a scoped GPU performance audit of a video-analytics workload is built to surface — and it is the difference between provisioning GPU capacity you need and paying for capacity a few CPU cores would have covered.
