A computer vision model that hits 98% accuracy in a demo is not a system that works in production. It is a system that works under the conditions the demo happened to capture — even lighting, framed subjects, a closed set of classes, and a frame rate the hardware comfortably sustains. Change any one of those, and the number that everyone agreed on in the meeting stops describing the system you actually deployed. This is the failure class that quietly defeats most production CV projects. The model doesn’t break loudly. It keeps emitting confident predictions — they just stop being correct, and nobody downstream can tell the difference between a confident right answer and a confident wrong one. We see this pattern regularly: the deployment passes acceptance, runs for weeks, and the first sign of trouble is an operator who has started ignoring the system because it cried wolf one time too many. Why Do CV Models That Pass in Demos Fail in Production? Demos are controlled environments disguised as representative ones. The lighting is what it was on the day. The camera angle is the one that looked good. The objects in frame belong to the classes the model was trained on, because the person running the demo chose them. None of those guarantees survive contact with a real deployment, and each one corresponds to a distinct failure mode. Lighting variability is the most underestimated. A classifier trained on daytime warehouse footage will see a completely different pixel distribution at dusk, under sodium lamps, or when a roller door opens and floods the frame with backlight. The convolutional features that fired reliably in training have no obligation to fire the same way on out-of-distribution illumination — and they frequently don’t. This is an observed pattern across the deployments we have worked on, not a benchmarked decay rate: accuracy that looked stable in validation drifts as the environment moves outside the lighting envelope the training set captured. Occlusion breaks the assumption that the object of interest is fully visible. A person behind a forklift, a product partially hidden on a shelf, two tracked entities that cross paths — these are routine in the real world and rare in curated training sets. Multi-object tracking is especially sensitive here, because an identity swap during occlusion propagates forward through every subsequent frame. Unknown-class flow is the failure mode nobody plans for. The model was trained on a closed set of categories, but production is an open world. When something the model has never seen enters the frame, a standard softmax classifier does not say “I don’t know.” It distributes probability across the classes it does know and reports the highest one with whatever confidence the geometry produces. That is silent misclassification, and it is the most dangerous of the four because it is invisible by construction. Edge throughput limits are the fourth, and they are a systems problem rather than a modeling one — covered in its own section below. The common thread: a model is a function fit to a distribution, and production is a different distribution that drifts continuously. Treating CV as a solved problem because the demo looked good is the same error as treating a benchmark score as a production guarantee — the reasons a GPU benchmark number diverges from real-workload behavior are structurally the same reasons a demo accuracy number diverges from production accuracy. How Do You Surface Unknown-Class Items Rather Than Misclassify Them? The fix for silent misclassification is architectural, not a matter of more training data. You cannot enumerate every object that will ever cross a camera, so the pipeline has to have an explicit answer for “this doesn’t match anything I was trained to recognize.” A few mechanisms, used together rather than in isolation: Confidence thresholding with calibration. Raw softmax outputs are not probabilities; modern networks are systematically overconfident. Temperature scaling or similar post-hoc calibration makes a confidence threshold meaningful enough to gate low-confidence predictions into a review queue instead of acting on them. Open-set recognition. Rather than forcing every input into a known class, an open-set head models the boundary of the known distribution and explicitly emits an “unknown” verdict for inputs that fall outside it. This is the difference between a closed-world classifier and a system honest about its own limits. Out-of-distribution detection. Energy-based scores or feature-space distance metrics flag inputs that are statistically unlike anything in training, independent of the classification head. Embedding-distance fallback. When a detection’s embedding is too far from every known class centroid, route the crop to a human or a slower verification path rather than committing to a label. The point is not which technique you pick. The point is that an unknown item must have somewhere to go other than the nearest known label. A modular pipeline that surfaces unknown-class flow converts a silent error into an actionable signal — the system reports “I am uncertain about this” instead of confidently mislabeling it. That single architectural decision is what separates a CV system an operator can trust from one they eventually learn to ignore. What Edge-Throughput Limits Break CV Pipelines at Deployment? The fourth failure mode is the gap between the frame rate the model can theoretically sustain and the frame rate the deployed hardware actually delivers under real load. On a workstation with an unloaded GPU, a detector might run at 60 FPS. On the edge device in the field — a Jetson-class module, an industrial PC with a constrained TDP, a camera with an embedded accelerator — that same model competes for memory bandwidth, shares the device with decode and preprocessing, and runs at a sustained throughput that is a fraction of the lab number. When throughput falls below the camera’s frame rate, the pipeline drops frames. Dropped frames break temporal continuity, which is exactly what multi-object tracking and action recognition depend on. A tracker that loses every third frame produces identity swaps and phantom tracks; an action recognizer fed a decimated clip sees a different action than the one that occurred. The model is correct frame-by-frame and wrong about the event. The mistake is benchmarking the model in isolation. The relevant number is sustained throughput of the whole pipeline — decode, preprocess, inference, postprocess, tracking — on the target device under realistic concurrent load, not the peak inference rate of the model on a clean bench. This is the same distinction that separates peak performance from steady-state behavior in any accelerated AI system: the transient peak tells you almost nothing about what the system holds under continuous production load. Establishing the real number requires empirical, workload-bound measurement on the deployment hardware, not a spec-sheet extrapolation. Practical levers when throughput is the binding constraint: Quantize to INT8 with TensorRT or an equivalent runtime, and validate that calibration didn’t quietly degrade accuracy on your edge conditions. Fuse the decode → preprocess → inference path so frames don’t bounce across the PCIe boundary or stall on host-side copies. Right-size the model: a smaller detector that sustains the frame rate beats a larger one that drops a third of the frames. Measure with the camera attached and the device thermally loaded, because sustained throughput under thermal throttling is the number that matters, not the cold-start burst. A Production CV Readiness Rubric Before treating a CV deployment as production-ready, score it against the four uncontrolled-environment failure modes. Each row is the question to answer, the failure if you can’t, and the evidence class of the check. Failure mode Readiness question Failure if unanswered Evidence class Lighting variability Was the model validated across the full illumination range of the site (dawn/dusk/artificial/backlight)? Accuracy drifts silently as conditions move outside the training envelope observed-pattern Occlusion Does tracking recover identity after partial/full occlusion without ID swaps? Propagating identity errors corrupt every downstream frame observed-pattern Unknown-class flow Does the pipeline emit an explicit “unknown” verdict for out-of-distribution inputs? Silent misclassification — confident wrong answers indistinguishable from right ones observed-pattern Edge throughput Is sustained whole-pipeline throughput on the target device ≥ camera frame rate under thermal load? Dropped frames break temporal continuity; tracking and action recognition fail benchmark (measure on target device) A deployment that can’t answer all four is not unready in the abstract — it has a named, specific gap, and the gap tells you exactly what to instrument before go-live. This rubric is the structure behind our computer vision practice readiness work: the assessment exists to turn “the demo looked great” into a defensible statement about how the system behaves on the conditions it will actually face. FAQ Why do CV models that pass in demos fail in production? Demos run under controlled conditions — even lighting, framed subjects, a closed set of known classes, and an unloaded device — that are not representative of deployment. Each of those conditions maps to a production failure mode: lighting variability, occlusion, unknown-class flow, and edge throughput limits. A model is a function fit to a distribution, and production is a different, continuously drifting distribution, so the demo accuracy number stops describing the deployed system. How do you surface unknown-class items rather than misclassify them? A standard softmax classifier has no way to say “I don’t know” — it forces every input into a known class and reports the highest one with misleading confidence. The fix is architectural: calibrated confidence thresholds, open-set recognition, out-of-distribution detection, and embedding-distance fallback give unknown inputs somewhere to go other than the nearest known label. This converts silent misclassification into an explicit “uncertain” signal an operator can act on. What edge-throughput limits break CV pipelines at deployment? When sustained whole-pipeline throughput on the target device falls below the camera’s frame rate, the pipeline drops frames. Dropped frames break temporal continuity, which causes identity swaps in multi-object tracking and misread events in action recognition. The relevant number is sustained throughput of the full decode-preprocess-inference-tracking path on the deployment hardware under realistic, thermally-loaded conditions — not the model’s peak rate on a clean bench. How do you architect a CV pipeline so failure is visible rather than silent? Build the pipeline so every failure mode has an explicit output rather than a quiet wrong answer: unknown-class detection emits an “unknown” verdict, low-confidence predictions route to a review queue, and throughput shortfalls are monitored against frame rate instead of assumed. A modular pipeline with explicit unknown-class surfacing fails visibly — it tells operators when it is uncertain — rather than emitting confident misclassifications that downstream systems trust as signal. What Makes This Worth Getting Right The cost of treating production computer vision as a solved problem is not a system that crashes — it is a system that keeps running while it is wrong, and a chain of downstream operators who trust its output as signal. Silent misclassification is expensive precisely because it is silent: by the time the error surfaces, decisions have been made on it. The same uncontrolled-environment failure class shows up wherever CV meets the real world — on a retail floor where scale changes the failure profile, and on industrial inspection lines where the lit, framed sample in the lab becomes a glare-streaked part moving past a camera at line speed. The discipline is the same in each: name the four failure modes, give each one an explicit output, and measure throughput on the device that will actually run it. A CV system earns production trust not by being accurate in a demo, but by being honest about the conditions under which its accuracy holds.
