“We built it on CUDA, so it runs anywhere with a GPU.” That sentence is true inside the data centre and false at the edge. The gap between those two statements is where most edge-agent deployments quietly break — not at build time, but at the production boundary, when the team discovers the accelerated code path has nowhere to run on the target device. CUDA is NVIDIA’s parallel-computing platform, and it is genuinely excellent at what it does: it lets you write kernels that run across thousands of GPU cores, and the surrounding ecosystem — cuDNN, TensorRT, NCCL — turns that into fast, well-tuned deep-learning execution. If your inference layer is a remote GPU with unbounded memory and a latency budget measured in seconds, CUDA is the right answer and the question barely needs asking. The trouble starts when you assume that answer generalises. A mobile assistant, an in-vehicle voice agent, or an air-gapped on-premise appliance does not have a CUDA-capable device underneath it. The moment part of the inference has to run on-device, the CUDA runtime is simply not present, and the code path you tuned so carefully cannot execute at all. What “CUDA applications” actually means in practice The phrase “CUDA applications” gets used loosely to mean “AI software that uses a GPU,” which is close enough to be misleading. Concretely, a CUDA application is a program that dispatches work to an NVIDIA GPU through the CUDA runtime and driver stack. That includes almost everything in the modern training pipeline — PyTorch and TensorFlow both compile down to CUDA kernels (via cuDNN and, increasingly, torch.compile and XLA) when a GPU is available — and it includes high-throughput server-side inference through TensorRT or the Triton Inference Server. What matters for an agent architect is that CUDA is a substrate, not a portable format. The dependency lives in the runtime and driver, not in your model weights. You can move a trained model’s weights anywhere; you cannot move the CUDA execution path anywhere. That distinction is the whole story. When people say “we’re CUDA-native,” they usually mean the accelerated numerical work — matrix multiplies, attention kernels, layernorm — is expressed in a way that only the CUDA stack can run efficiently. Inside a data centre this is invisible, because the substrate is always there. At the edge it becomes the load-bearing constraint. Where does CUDA belong in an agent’s inference stack? An agent’s compute has three tiers, and CUDA has a clear home in the first two. Training tier. This is CUDA’s strongest and least controversial territory. Multi-GPU training over NVLink, gradient synchronisation through NCCL, mixed-precision kernels tuned for HBM bandwidth — none of this has a serious portable alternative at scale. If you are training or fine-tuning, you are on CUDA, and you should be. Server-side inference tier. When inference runs on a GPU you control — a cloud endpoint, an on-premise inference server with real NVIDIA cards — CUDA through TensorRT remains the highest-throughput, lowest-cost-per-token option in most configurations we’ve worked through. Kernel fusion, graph optimisation, and INT8/FP8 calibration are all mature here. This is where the server tail of a hybrid agent lives. On-device / edge tier. This is where CUDA stops. A phone runs an Apple Neural Engine or an Adreno GPU; a browser exposes WebGL or WebGPU; an embedded automotive SoC may expose a vendor NPU with its own SDK. None of these speak CUDA. The accelerated path here has to be expressed through a portable runtime — CoreML on Apple silicon, ONNX Runtime with the appropriate execution provider across mixed hardware, or WebGL/WebGPU in the browser. The design error is not using CUDA. The error is failing to draw the line between tier two and tier three before you commit to an architecture. This is the same divergence point that invalidates data-centre agent-framework defaults at the edge, and it is why the edge-constrained data-centric approach for agent inference treats on-device portability as a first-class requirement rather than a porting task tacked on at the end. Why can’t a CUDA-accelerated path deploy to mobile, in-vehicle, or air-gapped targets? Because the CUDA runtime and driver are NVIDIA-specific system software, and none of those targets ship it. A CUDA kernel is not a portable artifact like an ONNX graph or a compiled WebAssembly module — it is machine work bound to a specific driver ABI and, ultimately, to NVIDIA silicon. There is no CUDA-on-CoreML shim and no CUDA-in-browser path. The dependency is total: either the device has an NVIDIA GPU with the driver stack, or the CUDA code does not run. This is worth stating plainly because “fallback” language hides it. Teams sometimes plan a “CUDA fallback” for edge, imagining they can degrade gracefully to CPU. But a fallback implies the same code with a slower backend; what actually happens at the edge is a different runtime entirely, with different operator coverage, different quantisation behaviour, and different numerical results. That is not a fallback — it is a second implementation. Discovering you need it at the production boundary means a rewrite, not a config change. In the edge engagements we’ve worked through, keeping the on-device tail-latency budget in the hundreds-of-milliseconds range depended on shipping a genuinely portable non-CUDA inference path — designed as such from the start (observed across TechnoLynx edge deployments; not a published benchmark, since the budget is workload-specific). A CUDA path with a CoreML afterthought bolted on late did not hit that envelope, because the model had never been shaped for the target runtime’s operator set. What replaces CUDA at the edge, and how do CoreML, ONNX Runtime, and WebGL relate? Think of the portable runtime as an inference adapter: a layer that takes your trained model and executes it on whatever accelerator the target actually has. The three you meet most often occupy different device niches. Runtime Primary target What it accelerates against When it fits CoreML Apple silicon (iOS, macOS) Apple Neural Engine, Metal GPU, CPU Native iOS/macOS agents; best power-per-inference on Apple devices ONNX Runtime Cross-platform (Windows, Linux, Android, embedded) Execution providers: CPU, DirectML, NNAPI, TensorRT, OpenVINO, others Mixed device fleets; one model graph, many backends via ONNX WebGL / WebGPU Browser GPU shaders exposed to JavaScript/WASM In-browser agents with no install; no native GPU access The unifying idea is that all three consume a portable model representation — ONNX being the most common interchange format — and map it onto the device’s real accelerator. ONNX Runtime is the pivot: because it supports pluggable execution providers, the same exported graph can target a TensorRT provider on a server GPU and an NNAPI provider on Android, which is exactly why it earns a place in a hybrid agent whose server tier is CUDA and whose edge tier is not. For anyone weighing the cross-vendor accelerator picture more broadly, OpenCL as a cross-platform edge inference path covers the layer beneath these runtimes on non-NVIDIA GPUs. Getting a model to run well through these runtimes is rarely just an export step. Operator coverage gaps, unsupported dynamic shapes, and precision drift under INT8 quantisation all surface here, which is why model optimization for edge inference — distillation, quantisation, and runtime fit treats runtime targeting as a design input rather than a packaging afterthought. How do you catch a CUDA-bound component in assessment, before it becomes a rewrite? The cheapest place to find a CUDA dependency is a design review; the most expensive is a shipped binary that will not deploy. A short diagnostic pass over each inference-touching component tells you which tier it can actually live in. CUDA-binding diagnostic checklist Does this component execute on the device tier? If it only ever runs server-side, CUDA is fine — stop here. If any part runs on-device, continue. Does it call a CUDA-only library at inference time? TensorRT, custom CUDA kernels, cuDNN-specific ops, or torch CUDA tensors in the inference path all bind you to NVIDIA silicon. Is there a clean ONNX/CoreML export? Try the export early. Missing or unsupported operators discovered now cost a model change; discovered at the boundary they cost a rewrite. Does the target runtime support the numerical precision the model needs? Quantisation behaviour differs across CoreML, ONNX Runtime providers, and WebGL. Verify accuracy on-target, not on the training GPU. Is the on-device latency budget met by the portable path, not the CUDA path? Measure the runtime you will actually ship, on hardware representative of the target. Two or more unresolved items on a device-tier component is the early-warning signal: that component is CUDA-bound and has no deployment story yet. Caught in assessment, the fix is a design revision — reshaping the model for a portable runtime. Caught in production, it is a rewrite of the accelerated path. The whole value of scoping CUDA explicitly is turning the second cost into the first. How this connects to the four edge-aware axes of the agent-framework decision CUDA scoping is not a standalone concern; it feeds directly into the broader edge-aware agent-framework assessment. The first axis of that decision — inference adapter portability — is precisely the CUDA-versus-portable-runtime split described here. Keeping that split explicit protects the other axes downstream: latency budgets stay honest because you measure the runtime you ship, deployment targets stay open because you never assume an NVIDIA device, and the compression-and-porting work has a clear boundary because you know exactly what stays a CUDA kernel on the server and what must be compressed and ported to CoreML, ONNX Runtime, or WebGL on-device. That server-versus-device compression boundary is a decision in its own right, and it is where GPU-tier optimisation and edge portability meet in our GPU and inference optimization work. None of this argues against CUDA. It argues for putting a line in the architecture — this is the server tier, this is the device tier — and defending it in review rather than discovering it at the boundary. FAQ What’s worth understanding about cuda applications first? A CUDA application dispatches numerical work to an NVIDIA GPU through the CUDA runtime and driver stack; frameworks like PyTorch and TensorFlow compile down to CUDA kernels (via cuDNN, TensorRT) when a GPU is present. In practice CUDA is a substrate bound to NVIDIA silicon, not a portable format — you can move model weights anywhere, but the CUDA execution path only runs where the runtime and driver exist. Where does CUDA belong in an agent’s inference stack — training, server-side inference, or on-device? CUDA belongs in the training tier and the server-side inference tier. Training over NVLink with NCCL and mixed-precision kernels has no serious portable alternative at scale, and server-side inference through TensorRT is usually the highest-throughput, lowest-cost-per-token path. On the device tier — phones, browsers, embedded SoCs — CUDA is absent and a portable runtime must take over. Why can’t a CUDA-accelerated inference path deploy directly to mobile, in-vehicle, or air-gapped edge targets? Because the CUDA runtime and driver are NVIDIA-specific system software that those targets do not ship. A CUDA kernel is bound to a specific driver ABI and NVIDIA silicon, so there is no CUDA-on-CoreML shim and no CUDA-in-browser path. What people call a “CUDA fallback” at the edge is really a second implementation on a different runtime, with different operator coverage and numerical behaviour. What replaces CUDA at the edge boundary, and how do CoreML, ONNX Runtime, and WebGL relate to it? They act as portable inference adapters that map a trained model onto the device’s real accelerator. CoreML targets Apple silicon, ONNX Runtime spans mixed fleets through pluggable execution providers, and WebGL/WebGPU runs in the browser. All three consume a portable model representation — ONNX being the common interchange — with ONNX Runtime acting as the pivot that can target a CUDA/TensorRT provider on the server and an NNAPI provider on Android from the same graph. How do I tell in assessment whether a component is CUDA-bound before it becomes a production-boundary rewrite? Run a short diagnostic on each device-tier component: check whether it calls CUDA-only libraries in the inference path, attempt a clean ONNX/CoreML export early, verify the target runtime supports the model’s precision, and measure latency on the runtime you will actually ship. Two or more unresolved items on a device-tier component means it is CUDA-bound with no deployment story yet. Caught in assessment the fix is a design revision; caught in production it is a rewrite. How does the CUDA-versus-portable-runtime split connect to the four edge-aware axes of the parent agent-framework decision? The split is the first axis — inference adapter portability. Keeping it explicit protects the other axes: latency budgets stay honest because you measure the shipped runtime, deployment targets stay open because you never assume an NVIDIA device, and the compression-and-porting work has a defined boundary between server-side CUDA kernels and on-device CoreML/ONNX/WebGL execution. A useful discipline is to end every architecture review of an edge-bound agent with one question: for each component that touches inference, name the tier it runs on and the runtime that will execute it there. Any component whose answer is “CUDA, and we’ll figure out the edge later” is the one that will fail at the boundary — the portable inference adapter is not a porting task, it is part of the design.
