ML Model Deployment Tools: Shipping Classical and Deep CV Stages to Production

A production computer vision pipeline is almost never one deep model behind an endpoint. It is a chain: a region-of-interest crop, an ORB or SIFT feature pass, maybe a HOG descriptor, and then the CNN or ViT that everyone thinks of as “the model.” When teams shop for ML model deployment tools, they picture wrapping that one deep model — export it, expose an endpoint, done. The chain is where that picture breaks.

The divergence point is simple and expensive to discover late: classical feature-extraction stages are not framework-serializable model artifacts. They are code. Tools like NVIDIA Triton Inference Server, TorchServe, ONNX Runtime, and BentoML are built around the assumption that you hand them a model — a graph with weights — not an OpenCV preprocessing routine that calls cv2.goodFeaturesToTrack and branches on the result. A serving runtime has nowhere to put that logic. Teams that recognise this and pick tools per stage ship reliably. Teams that assume one tool covers the whole pipeline find the gap in production, when the notebook prototype refuses to reproduce on the target hardware.

Why “wrap the model” fails for hybrid pipelines

The mental model behind most deployment tooling is a single differentiable graph. You trace a PyTorch module, export to ONNX or TorchScript, and the serving layer loads it, batches requests, and runs inference on GPU. That path is genuinely good — it is the mature, well-trodden part of the ecosystem. The problem is that it only covers the deep stage.

A classical stage does not fit that shape. SIFT keypoint extraction, ORB descriptors, HOG feature computation, contour-based ROI cropping — these are imperative algorithms with data-dependent control flow. They do not export to ONNX because there is no static computation graph to export; the number of keypoints, the branching on image content, the CPU-side memory allocation all happen at runtime. You cannot hand cv2.ORB_create() to Triton and expect it to serve.

What teams do instead, when they haven’t planned for this, is one of three bad things. They rewrite the classical stage as an approximate neural network so it fits the serving tool — losing accuracy and the compute advantage that made the classical stage worth having. They keep the classical stage as a Python script glued in front of the server with a queue, and inherit a second runtime with its own dependency drift. Or they push everything into one Docker container running a Flask app, and lose the batching, versioning, and GPU-scheduling benefits the serving tool exists to provide. Each of these is a symptom of forcing a uniform path onto a non-uniform pipeline.

The correct frame is that a hybrid pipeline has at least two deployment problems, and they have different right answers. The deep stage wants a model server. The classical stage wants a well-built, reproducible container with pinned OpenCV and a clean interface. Treating them as one problem is what produces the prototype-to-production gap.

What “ml model deployment tools” actually covers

The phrase bundles several distinct jobs that are worth separating before you choose anything:

Model serialization / export — turning a trained network into a portable artifact. ONNX, TorchScript, and SavedModel live here. This is a model concern and does not touch classical code.
Inference runtime — the engine that executes the exported graph efficiently on target hardware. ONNX Runtime, TensorRT, and OpenVINO are runtimes. A model compiler lowers the graph to hardware-specific kernels here.
Model serving — the process manager around the runtime: request batching, versioning, multi-model hosting, health checks. Triton, TorchServe, and BentoML operate at this layer.
Packaging / orchestration — the container and its deployment surface. Docker, Kubernetes, and the CI build step that turns your repository into a reproducible image.

Classical CV code slots into serialization-not-applicable, runtime-is-just-CPU-or-OpenCL, and packaging-is-everything. The mismatch that trips teams up is that the industry’s loudest tooling — serving platforms — solves a layer that classical stages don’t need, while under-serving the packaging layer that classical stages depend on entirely.

Which tools fit which stage?

The following is a decision surface, not a ranking. The right tool depends on what the stage is, not on which tool is most popular.

Stage characteristics	Fitting tool class	Concrete example	Why
Deep model, GPU target, high request volume	Model server + GPU runtime	Triton + TensorRT	Dynamic batching and multi-model hosting justify the operational weight
Deep model, single model, moderate volume	Lightweight server	TorchServe or BentoML	Simpler ops; no need for Triton’s multi-backend machinery
Deep model, CPU service, cross-platform	Portable runtime in a container	ONNX Runtime or OpenVINO	No GPU to schedule; the runtime is the deployment
Classical feature stage (SIFT/ORB/HOG/ROI)	Containerized code path	Docker image, pinned OpenCV, gRPC/REST interface	Not a servable model; packaging and reproducibility are the whole job
Classical stage on a low-power edge device	Compiled native binary	C++/OpenCV or OpenCL build	Python runtime overhead is unacceptable; the compile target constrains everything

Two rows deserve emphasis. When a stage is a lone deep model on a CPU service, a full serving platform is usually overkill — ONNX Runtime inside a plain container gives you portable, versioned inference without the operational surface of Triton. And the classical rows never touch a serving platform at all; the useful engineering there is a disciplined build, not a model host. If you are serving one CV stage and reaching for Kubernetes plus a multi-model server out of habit, that is worth questioning before it becomes your on-call burden.

What actually changes moving from notebook to production

The gap between a Python/OpenCV prototype and a deployed runtime is where hybrid pipelines silently break, and almost none of it is about the deep model. A few patterns recur across the deployment work we do:

Preprocessing determinism is the first casualty. A notebook resizes with whatever OpenCV interpolation default was in scope, normalizes with a mean the training author remembers and nobody wrote down, and reads images in BGR because that is what cv2.imread returns — while the model was trained on RGB. Ported naively, the deployed pipeline runs, produces plausible-looking outputs, and is quietly wrong. This is an observed pattern across CV deployments, not a benchmarked failure rate: the single most common production discrepancy we see traces to a preprocessing detail that was never pinned.

Dependency and version drift is the second. The notebook ran on OpenCV 4.8 with a particular NumPy; the container built six weeks later pulled 4.9, and ORB’s default parameters shifted just enough to change the keypoint set feeding the matcher. Pinning every classical dependency in the container image — the same way you pin the model artifact version — is the fix, and it is a build-step concern, not a serving-tool concern.

Hardware divergence is the third, and it is where the cross-discipline seam sits. A pipeline validated on a workstation GPU behaves differently once the classical preprocessing runs on an edge CPU or an OpenCL device. Which deployment tools and runtimes are even viable is constrained by the target: deploying low-power classical preprocessing on edge devices narrows the field before you consider serving at all. The choice of compiler flags for the target runtime can change numerical behaviour subtly enough to matter for a matching threshold downstream.

Handled well, the porting step turns from weeks of interactive debugging into a repeatable build. That is the ROI: not a faster model, but a pipeline whose deployed behaviour matches its validated behaviour on the first try.

How tool choice preserves the compute savings of a hybrid pipeline

The reason to keep classical stages in the first place is efficiency. Using a cheap ORB or HOG pass to reject the easy negatives, or a classical ROI crop to shrink what the deep model sees, is what keeps a pipeline affordable. In configurations we’ve worked with, front-loading classical filtering ahead of the deep stage cuts the compute the neural network has to do by roughly 3–10× — an observed pattern whose exact figure depends entirely on how much of the input a cheap stage can discard, not a published benchmark.

That saving is a pipeline property, and it only survives to production if each stage is deployed on tooling that respects its cost profile. Force the classical stage into a neural approximation so it fits Triton, and you have replaced a near-free CPU operation with a GPU forward pass — the saving is gone before the pipeline ships. Keep the classical stage as compiled or containerized code and serve only the deep stage on the model server, and the 3–10× advantage reaches production intact. The same logic explains where classical preprocessing still earns its place ahead of a YOLO detector: the deployment decision and the architecture decision are the same decision, seen at two points in time.

This is why deployment-tool selection is properly an output of a computer vision architecture review, not a task you do at the end. The review decides which stages stay classical code and which become served models; the tool choice per stage falls out of that split. Deciding tooling before the split is decided is how teams end up retrofitting a serving platform around code that was never a model.

FAQ

How should you think about ml model deployment tools in practice?

“ML model deployment tools” bundles several distinct jobs: serialization (ONNX, TorchScript), inference runtime (ONNX Runtime, TensorRT), model serving (Triton, TorchServe, BentoML), and packaging/orchestration (Docker, Kubernetes). In practice you rarely use one tool for a whole CV pipeline — you match a tool to each stage’s job, because serving platforms assume a servable model graph, which a classical OpenCV stage is not.

Which ML model deployment tools fit a hybrid CV pipeline where classical feature stages sit alongside a deep model?

Deploy the deep stage on a model server matched to its target — Triton plus TensorRT for high-volume GPU, TorchServe or BentoML for a single model, ONNX Runtime in a container for a CPU service. Deploy the classical feature stage as containerized or compiled code with pinned OpenCV, because it is not a serializable model artifact and no serving platform will package it.

How do you deploy a classical OpenCV feature-extraction stage that a model-serving tool won’t package?

Treat it as a code deployment, not a model deployment: build a reproducible container with pinned OpenCV and dependency versions, expose a clean gRPC or REST interface, and on low-power edge targets compile it as a native binary instead of shipping a Python runtime. The whole job is packaging and reproducibility, since there is no static graph to serialize.

What changes when moving a Python/OpenCV prototype to a deployable production runtime (ONNX, Triton, TorchServe, BentoML)?

Three things bite most: preprocessing determinism (interpolation, normalization means, BGR vs RGB channel order that was never written down), dependency and version drift in the classical code, and hardware divergence from workstation GPU to edge CPU or OpenCL. Pinning every classical dependency and fixing preprocessing exactly turns porting from weeks of debugging into a repeatable build step.

How do you match deployment tools to targets — cloud GPU, CPU service, or a low-power edge device?

The target constrains the viable tools before serving is even considered. Cloud GPU with high volume justifies Triton plus TensorRT; a CPU service is served well by ONNX Runtime or OpenVINO in a plain container; a low-power edge device usually needs compiled native code because a Python runtime’s overhead is unacceptable there.

When is a full model-serving platform overkill versus a simple containerized inference path for a CV stage?

A single deep model at moderate volume on a CPU service rarely needs a multi-model serving platform — ONNX Runtime inside a container gives portable, versioned inference without Triton’s operational surface. Reach for a full serving platform when dynamic batching, multi-model hosting, and GPU scheduling genuinely justify the on-call weight.

How do deployment-tool choices preserve the compute savings of splitting a pipeline into classical and deep stages?

The 3–10× compute saving from cheap classical filtering is a pipeline property that only survives if each stage is deployed on tooling that respects its cost profile. Keep the classical stage as compiled or containerized code and serve only the deep stage on the model server; forcing the classical stage into a neural approximation to fit one serving tool replaces a near-free CPU operation with a GPU forward pass and erases the saving before you ship.

The uncomfortable part is that the deployment-tool question and the pipeline-architecture question are the same question asked at two moments. If you cannot yet say which stages of your pipeline are classical code and which are served models, no list of deployment tools will resolve it for you — that split is the decision, and the tooling is downstream of it.