Why the edge matters for computer vision A cloud-based computer vision pipeline works like this: a camera captures an image, the image is transmitted over the network to a cloud server, the server runs inference, and the result is transmitted back. The round-trip latency — image transmission, queuing, inference, result transmission — is typically 100–500 milliseconds, sometimes more under network congestion. For many applications, that latency is acceptable. For others — industrial inspection at production line speed, autonomous navigation, real-time safety monitoring — it is not. Edge deployment moves the inference step from the cloud to a device co-located with the camera: an NVIDIA Jetson module, a Google Coral accelerator, a Qualcomm AI-optimised SoC, or an Intel Neural Compute Stick attached to an embedded system. The image never leaves the device. Inference latency drops to 10–50 milliseconds. Network bandwidth requirements drop to near zero (only results, not images, are transmitted). And the system continues operating when the network connection is unavailable — which in industrial environments happens more often than IT architecture diagrams suggest. The trade-off: edge devices have constrained compute, memory, and power budgets compared to cloud servers. A model that runs comfortably on an NVIDIA A100 in the cloud may not run at all on a Jetson Nano, and the model modifications required to fit the edge hardware’s constraints affect accuracy, throughput, or both. How do you fit a production model onto an edge device? As an illustrative example from our edge-deployment engagements (an observed range, not a benchmarked industry rate): a ResNet-152 trained for image classification has approximately 60 million parameters, requiring roughly 240 MB of memory and significant compute for each inference pass. An edge device with 2–4 GB of shared RAM and a low-power GPU or NPU cannot run this model at production frame rates. The model must be made smaller, faster, or both — without degrading accuracy below the application’s acceptance threshold. Quantisation reduces model precision from 32-bit floating point to 16-bit or 8-bit integer representation. INT8 quantisation typically reduces model size by 4× and improves inference speed by 2–4×, with accuracy degradation of 0.5–2 percentage points for well-quantised models. Post-training quantisation (applying quantisation to an already-trained model) is the simplest approach; quantisation-aware training (training the model with quantisation constraints) produces better accuracy preservation but requires access to the training pipeline. TensorRT, OpenVINO, and TFLite all support quantisation workflows for their respective hardware targets. Model architecture selection. Not all model architectures quantise equally well, and not all architectures are designed for edge deployment. MobileNet, EfficientNet-Lite, and YOLO-NAS are architectures explicitly designed for resource-constrained inference — in our experience across edge-CV engagements, they achieve competitive accuracy with 5–20× fewer parameters than their full-scale equivalents (an observed range, not a benchmarked industry rate). Choosing an edge-optimised architecture from the start avoids the lossy compression of shrinking a large model to fit a small device. Knowledge distillation trains a small “student” model to reproduce the outputs of a large “teacher” model. The student model inherits the teacher’s learned representations at a fraction of the parameter count and computational cost. This approach is particularly effective when the large model achieves accuracy that the edge application needs, but the architecture is too large for edge deployment — the distillation process transfers the accuracy into a deployable form factor. We regularly apply these techniques in combination: selecting an edge-optimised architecture, training with quantisation awareness, and distilling from a larger model when the accuracy-size trade-off requires it. The specific combination depends on the edge hardware target and the application’s latency and accuracy requirements. Hardware selection: matching compute to workload Edge AI hardware spans a wide range of capability, power consumption, and cost. The selection criteria are workload-specific: NVIDIA Jetson family (Orin Nano, Orin NX, AGX Orin) provides CUDA-compatible GPU compute at the edge, supporting the full NVIDIA inference stack (TensorRT, DeepStream). The Jetson platform is the most capable edge AI hardware available, with the AGX Orin delivering up to 275 TOPS of AI compute. The trade-off is power consumption (15–60W depending on the module) and cost (£200–£1,500 per module). For applications that require high throughput (multiple camera streams, high-resolution processing, complex models), Jetson is typically the right choice. Google Coral (USB Accelerator, Dev Board, M.2 module) provides a dedicated Edge TPU that accelerates TFLite models at very low power (2–4W). The performance ceiling is lower than Jetson — the Edge TPU supports a specific set of operations optimised for MobileNet-class models — but the power and cost profile (£50–£150 per unit) makes it suitable for high-volume deployments where per-unit cost matters. Qualcomm and MediaTek AI SoCs integrate neural processing units into mobile and IoT system-on-chip designs. These are the foundation of AI capability in smartphones, smart cameras, and consumer IoT devices. The advantage is integration density and power efficiency; the constraint is software ecosystem maturity and model compatibility. The GPU performance considerations that apply to cloud inference also apply to edge inference, with the additional constraint that edge devices do not have the thermal headroom or memory bandwidth of data centre hardware. Memory bandwidth, in particular, is often the binding constraint on edge devices — a model that is compute-bound on cloud hardware may become memory-bandwidth-bound on edge hardware, requiring different GPU inference latency optimisation strategies. Deployment pipeline considerations Deploying a model to an edge device is not the same as deploying a model to a cloud server. The operational constraints are different, and the deployment pipeline must account for them. Over-the-air model updates. Edge devices in the field need to receive model updates without physical access. This requires an update mechanism that downloads the new model, validates it (checksum, inference test on reference data), and swaps it atomically — so that a failed update does not leave the device without a functioning model. The update bandwidth is constrained — in our experience across edge engagements (a planning heuristic, not a benchmarked industry rate): a 50 MB quantised model over a cellular connection is feasible; a 500 MB full-precision model is not. Fallback and degradation handling. What happens when the model fails to load, the inference engine crashes, or the device runs out of memory? Cloud deployments handle this with redundancy — another instance picks up the load. Edge deployments must handle it locally: a fallback model (simpler, smaller, less accurate but always available), a degraded-mode protocol (pass through without inference, alert the monitoring system), or a restart-and-recover process that restores the device to a known-good state. Monitoring and telemetry. Edge devices produce monitoring data — inference latency, prediction distributions, error counts, device temperature, memory utilisation — that must be transmitted to a central monitoring system. The telemetry pipeline must be lightweight (the device’s compute budget is consumed by inference, not monitoring), resilient to connectivity interruptions (buffer and forward when the connection is restored), and structured for anomaly detection (so that a device that is behaving differently from its peers is flagged automatically). Edge CV deployment: pilot vs scale-out checklist Dimension Pilot (1–5 devices) Scale-out (tens to hundreds of devices) Hardware selection Single platform — e.g. Jetson Orin NX for flexibility during model iteration Mixed fleet matched to workload: Jetson for multi-stream sites, Coral or Qualcomm SoCs where per-unit cost and power (2–4 W) dominate Model optimisation level Post-training INT8 quantisation via TensorRT or TFLite; edge-optimised architecture (MobileNet, EfficientNet-Lite, YOLO-NAS) Quantisation-aware training + knowledge distillation; per-hardware model variants compiled against each target’s inference engine Monitoring infrastructure Basic telemetry — inference latency, error counts, device temperature — forwarded to a central dashboard on reconnect Full anomaly-detection pipeline: buffered telemetry with store-and-forward, peer-comparison alerts, prediction-distribution drift detection Update mechanism Manual model push or scripted SCP/SSH; validate with checksum and reference-data inference test OTA update service with atomic model swap, automatic rollback on validation failure, bandwidth-aware scheduling (≤ 50 MB quantised payloads over cellular) Redundancy Restart-and-recover process that restores a known-good model state; no hardware redundancy Fallback model per device (simpler, always loadable), degraded-mode protocol (pass-through + alert), spare-device pool for field swap Validation scope Accuracy check on a reference dataset before deployment; manual review of edge-case predictions Full accuracy regression against production distribution, automated A/B comparison between old and new model, per-site acceptance gates before fleet-wide rollout When edge deployment is the right architecture In our experience, edge deployment is justified when one or more of these conditions hold: the latency requirement is below what cloud inference can reliably deliver, the bandwidth cost of transmitting images to the cloud exceeds the cost of edge compute, the system must operate during network outages, or data privacy requirements prohibit transmitting images off-premises. When none of these conditions hold, cloud inference is usually simpler, more flexible, and easier to maintain. The edge vs cloud question is an architecture decision, not a technology preference — and getting it wrong in either direction has cost and capability consequences. A Production CV Readiness Assessment includes edge-specific hardware, model optimisation, and deployment architecture analysis for teams making this decision.
