The demo worked perfectly

In our experience across CV deployments, the object detection model scored 94% mAP on the evaluation dataset — an operational measurement from one such project. The integration test passed. The stakeholder demo was clean: bounding boxes appeared where they should, confidence scores were high, and the engineering team felt ready to deploy. Four weeks into production, the false-positive rate was three times higher than testing predicted, the model missed an entire class of defect it had never encountered in training, and the operations team was spending more time managing the model’s errors than they had spent on the manual process it replaced.

This is not an unusual outcome. It is the expected outcome when an off-the-shelf model — YOLO, Faster R-CNN, EfficientDet, or any pre-trained detection architecture from the PyTorch or TensorFlow ecosystems — is deployed into a production environment that differs from its training conditions in ways that benchmark evaluation does not measure. The failure is not in the model architecture. The failure is in the assumption that benchmark accuracy transfers to production reliability.

Where does the accuracy gap come from?

The gap between benchmark performance and production performance has specific, identifiable causes. Understanding these causes is the difference between diagnosing a deployment failure retroactively and preventing it structurally.

Lighting and environmental variation. Benchmark datasets are typically captured under controlled conditions — consistent lighting, stable backgrounds, uniform image quality. Production environments are not controlled in the same way. A warehouse camera operates under fluorescent lighting that shifts colour temperature across the day. An outdoor surveillance system contends with shadows, glare, weather, and seasonal lighting changes. A manufacturing inspection station has lighting that degrades as bulbs age. Each of these variations introduces a distribution shift between training data and production data, and the model’s accuracy degrades proportionally to the magnitude of that shift, often without any visible error signal until someone audits the results.

Class distribution mismatch. Benchmark datasets are typically class-balanced: roughly equal numbers of examples per category, or at least a distribution representative of the evaluation task. Production environments are rarely class-balanced. Across our manufacturing CV engagements, 97–99% of units are defect-free — the positive class (defect present) is extremely rare. That figure is an observed range across our engagements, not a benchmarked industry rate. A model trained on a balanced dataset will produce a different precision-recall trade-off in production than it showed during evaluation, because the base rate of the positive class has changed by an order of magnitude. The practical consequence: a false-positive rate that was acceptable at 1% in evaluation becomes operationally problematic when applied to millions of units per month (also an observed pattern across our CV engagements, not a benchmarked industry rate).

Domain-specific failure modes. Every deployment domain has failure classes specific to its operational context — and that off-the-shelf models have never seen. A retail shelf monitoring system encounters products that partially occlude each other, promotional displays that change the visual context weekly, and product packaging redesigns that change the appearance of items the model was trained to recognise. A medical imaging system encounters imaging artifacts, patient positioning variations, and pathology presentations that differ from the training distribution. These are not edge cases. In our experience, they are the normal operating conditions of the specific domain, and they are invisible to a model trained on a generic or cross-domain dataset.

Why testing on a held-out set does not catch these failures

The standard ML evaluation methodology — train on one portion of the dataset, evaluate on a held-out portion — measures a model’s ability to generalise within the training distribution. It does not measure the model’s ability to generalise to a different distribution, which is exactly what production deployment requires.

A held-out test set drawn from the same dataset as the training data shares the same lighting conditions, the same class distribution, the same domain characteristics, and the same failure modes. Evaluating on this set tells you how well the model has learned the dataset. It does not tell you how the model will behave when the camera angle changes, the lighting shifts, the product mix evolves, or a defect type appears that was not represented in the training data.

We encounter this pattern regularly: a team evaluates a model on a held-out set, reports strong metrics, deploys to production, and discovers that the production accuracy is 10–20 percentage points below the evaluation accuracy (an observed pattern across our CV engagements where training data did not match production conditions, not a benchmarked figure). The team’s first instinct is usually to retrain with more data or try a different architecture. In our experience, the more productive first step is to characterise the distribution gap between training data and production data — because the gap, once identified, often reveals specific correctable causes (lighting normalisation, class rebalancing, domain-specific augmentation in PyTorch or Albumentations pipelines) rather than requiring a wholesale model replacement.

What does production-grade evaluation actually require?

Moving from benchmark evaluation to production evaluation requires testing against the actual conditions of deployment, not against a subset of the training distribution.

Environment-representative test data. The evaluation dataset must be captured from the production environment — same cameras, same lighting, same operating conditions, same class distribution. If the production environment changes across shifts, seasons, or product cycles, the evaluation dataset must include samples from each variant. This is more expensive to construct than a curated benchmark dataset, but it is the only evaluation approach that predicts production performance.

Domain-specific metrics. Overall accuracy and mAP are useful for architecture comparison but insufficient for production decision-making. Production evaluation requires metrics that map to operational impact: false-positive rate at the operating threshold (how many good items will be incorrectly flagged?), false-negative rate per defect class (which defect types will be missed?), performance across data subsets (does the model degrade for specific product variants, lighting conditions, or time periods?), and latency under production load (can the model maintain throughput at line speed, particularly when deployed through TensorRT or ONNX Runtime on edge hardware?). These metrics are not exotic. They are the questions the operations team will ask after deployment, and answering them before deployment prevents the discovery phase from happening in production.

Out-of-distribution behaviour characterisation. What happens when the model encounters an input it was not trained on? Does it assign a low confidence score (desirable — the system can flag uncertain cases for human review) or a high confidence score on an incorrect class (dangerous — the system fails silently)? Characterising this behaviour before deployment requires deliberately testing with inputs that fall outside the training distribution — novel objects, adversarial lighting, corrupted images, occluded views. The model’s behaviour on these inputs determines whether it fails safely or fails silently, which is the difference between a production system that degrades gracefully and one that produces undetected errors.

The quality control workflows that integrate AI and computer vision depend entirely on this production-grade evaluation. A model that has not been evaluated against production conditions is a model whose production failure rate is unknown — not zero, unknown.

A production-readiness checklist

Before a CV model crosses from staging to production, the team should be able to answer each of these with a documented artifact, not an assumption.

Check	What “ready” looks like	Common failure mode
Test data provenance	Captured from production cameras under the full range of operating conditions	Test data is a held-out slice of the same curated training set
Class distribution	Production base rates reflected in evaluation; thresholds tuned to that distribution	Balanced eval set; thresholds tuned to balanced data
Per-class error metrics	False-positive and false-negative rates reported per class, per environment	Single aggregate accuracy or mAP figure
Throughput under load	Latency p50/p95 measured at production batch size on production hardware	Latency measured on a developer workstation
OOD behaviour	Confidence distribution characterised on inputs outside the training set	OOD inputs untested; silent high-confidence errors possible
Drift monitoring	Logging in place for input distribution and confidence drift post-deployment	Model deployed without any production-side observability

The checklist is intentionally boring. Each row corresponds to a class of production surprise we have seen often enough to expect.

When is fine-tuning enough versus replacing the model?

Off-the-shelf models are useful starting points. Transfer learning from pre-trained architectures (ResNet, EfficientNet, Vision Transformers, DINOv2 backbones) reduces training time and data requirements. The decision between fine-tuning and replacement is not about the architecture itself; it is about whether the feature representations the backbone learned on ImageNet or LAION-scale data are relevant to the production domain.

Fine-tuning is usually enough when the production domain shares low-level visual statistics with the pre-training data — natural images, similar object scales, similar lighting regimes. Fine-tuning is rarely enough when the production domain has fundamentally different statistics: thermal imagery, medical modalities, microscopy, or synthetic-aperture data. In those cases, a backbone trained on the right modality (or trained from scratch on a large in-domain dataset) outperforms a fine-tuned generic backbone by margins that fine-tuning cannot close.

The intermediate case is the most common: the domain is natural imagery, but with strong domain-specific structure (retail shelves, manufacturing lines, agricultural fields). Here, fine-tuning a generic backbone with domain-specific augmentation and class rebalancing is usually the right starting point — and the question of whether to go further is answered by the production-grade evaluation above, not by intuition.

The production-readiness question

The decision to deploy a computer vision model is not a binary pass/fail on a benchmark. It is an assessment of whether the model, the data pipeline, the deployment infrastructure, and the monitoring systems are collectively ready to operate reliably under production conditions — with known and documented performance characteristics, not aspirational ones.

Off-the-shelf models are useful starting points. The failure is not in using these architectures. It is in deploying them without production-representative evaluation, without domain-specific fine-tuning, and without monitoring infrastructure that detects when production conditions drift away from training conditions. The gap between evaluation-set accuracy and production reliability is where most CV deployment surprises originate — a Production CV Readiness Assessment quantifies that gap before it becomes an operational cost.

FAQ

Why do off-the-shelf computer vision models fail in production?

Because benchmark accuracy is measured under the dataset’s own distribution, not the production distribution. Off-the-shelf detectors (YOLO, Faster R-CNN, EfficientDet) degrade systematically when lighting, class base rates, occlusion patterns, or domain context differ from training conditions. The failure is structural, not edge-case rarity.

What kinds of edge cases break public detection and classification models in real deployments?

Lighting drift across shifts and seasons, extreme class imbalance (often 97–99% of items belonging to one class in QC contexts), partial occlusion, product or scene redesigns that shift visual context, and domain-specific artefacts that were never represented in the training data. These are normal operating conditions for the deployment, not statistical outliers.

How do I test a CV model against production data before shipping it?

Capture evaluation data from production cameras under the full range of operating conditions, measure per-class false-positive and false-negative rates rather than aggregate mAP, test latency at production batch size on production hardware, and deliberately probe out-of-distribution behaviour to confirm the model fails with low confidence rather than silently mislabelling.

What does it cost to discover an off-the-shelf model is wrong only after deployment?

The cost is typically the operational overhead of managing the model’s errors — false positives reviewed by humans, false negatives that escape into downstream processes — plus the rework of building production-representative evaluation that should have existed before deployment. In our experience the operational cost dwarfs the cost of doing production-grade evaluation up front.

When is fine-tuning enough versus replacing the model entirely?

Fine-tuning a pre-trained backbone is usually enough when the production domain shares low-level visual statistics with the pre-training data. Replacement (or a backbone trained on the right modality) is needed when the domain is fundamentally different — thermal, medical, microscopy, synthetic-aperture. The production-grade evaluation, not intuition, decides which case applies.

Which object-detection problems are inherent to the model class versus solvable with more data?

Distribution shift, class imbalance, and domain-specific failure modes are usually solvable with the right data — production-representative samples, rebalanced thresholds, domain-specific augmentation. Problems inherent to the model class are rarer: hard occlusion reasoning, small-object detection at extreme scales, and fine-grained discrimination that exceeds the backbone’s representational capacity. Those typically need an architectural change, not more data.