Heterogeneous Architecture for Inference: CPU, GPU, and WASM Targets in Practice

“We’re rewriting the inference path in C++.” It is one of the most common sentences we hear from a platform team that has just decided a Python service is too slow, and it is usually the wrong frame. The question is not which language the path should live in. It is which compute target each stage should run on.

That distinction sounds pedantic until you watch a full rewrite land and the latency barely move. A team spends a quarter porting a preprocessing-heavy path to native C++, only to discover the bottleneck was always the model forward pass on an under-fed GPU — a stage that was never going to get faster from a language change. The port relocated the code without removing the cost. Heterogeneous architecture for inference is the discipline that prevents this: it treats the inference path as a system where each stage can land on a different target, matched to where the profiled bottleneck actually lives.

What does heterogeneous architecture mean for an inference path?

An inference path is rarely one homogeneous block of work. In practice it decomposes into at least three kinds of stages: model compute (the forward pass, attention kernels, convolutions), pre- and post-processing (tokenisation, image decode, resize, NMS, formatting), and IO (loading weights, moving tensors across the PCIe bus, reading and writing request data). Each of these has a different performance profile, and each has a compute target where it runs best.

Heterogeneous architecture means you stop asking “where should the path live?” and start asking “where should each stage live?” The candidate targets, in the porting context we work in most often, are three:

GPU / CUDA — massively parallel, high arithmetic throughput, but with a fixed cost to move data in and out. Best for the model forward pass and any genuinely data-parallel transform.
Native CPU C++ — low per-call latency, tight control over memory layout and SIMD, no host-device transfer. Best for branchy pre/post-processing and small tensors where kernel-launch overhead would dominate on the GPU.
WASM / Pyodide — the model runs inside the browser or an edge sandbox, with no server round trip. Best when the deployment constraint is “no backend” or “run on the client,” and the model is small enough to fit the target’s memory and compute envelope.

The core claim here is simple and it is the one most single-language ports violate: a heterogeneous mapping grounded in profiling routes each stage to the target that moves its bottleneck, whereas a uniform port assumes one target fits the whole path and often relocates cost without removing it. We treat that as the default hypothesis on any port-assessment engagement, and it holds far more often than teams expect.

How do you decide which target each stage runs on?

The decision is driven by per-stage bottleneck attribution, not by a general preference for one target. Before any mapping is committed, the path is profiled so that the total latency is split into named contributions: how much time is spent in model compute, how much in Python-level overhead, how much waiting on IO. Our companion piece on profiling the Python inference path before a C++ or WASM port covers how that attribution is actually captured; this article is about what you do with the numbers once you have them.

The mapping rule is: assign each stage to the target that removes the bottleneck that stage owns. A stage that is bound by Python interpreter overhead and object churn is a candidate for native C++ (or Cython) — moving it to the GPU does nothing if the work is not data-parallel. A stage bound by arithmetic on large tensors belongs on the GPU, and staying in optimised CPU code will leave throughput on the table. A stage bound by IO is not fixed by any compute-target change; it is fixed by batching, prefetching, or memory-layout work, and porting it is wasted effort.

Target-assignment decision table

Match the profiled bottleneck class in the left column to the target that actually moves it. This is the surface we hand teams once profiling is done.

Profiled bottleneck (per stage)	Recommended target	Why it moves the cost	Common trap
Model forward pass, large tensors, data-parallel	GPU / CUDA	Arithmetic throughput; kernel fusion via TensorRT or `torch.compile`	Small tensors where launch overhead dominates
Python interpreter overhead, object churn, branchy logic	Native CPU C++ (or Cython)	Removes interpreter cost; SIMD via AVX; no host-device copy	Assuming GPU helps non-parallel work
Host-device transfer / PCIe traffic	Neither — batching / layout fix	Cost is data movement, not compute	Porting the stage instead of the data flow
“No backend” / client-side deployment constraint	WASM / Pyodide	Eliminates the server round trip entirely	Model too large for the WASM memory/compute envelope
Small pre/post transform between two GPU stages	Keep on GPU or fuse	Avoids a round trip back to host	Bouncing to CPU for a trivial op

The evidence class behind these mappings is observed-pattern: they are the routing decisions we see pay off repeatedly across porting engagements, not a single published benchmark. The right assignment for your path still depends on your profile — the table tells you what to look for, not what your numbers will say.

When does splitting the path beat porting it all to one target?

Splitting wins when the profiled bottlenecks live in different stages that want different targets. If your attribution shows 70% of latency in the model forward pass and 25% in a Python tokeniser, a uniform “rewrite everything in C++” port fixes the wrong quarter and leaves the dominant 70% on a CPU that cannot compete with a GPU on that work. The correct move is heterogeneous: model compute to the GPU, tokeniser to native C++ or a fused CPU stage, and leave the IO alone unless it is measurably in the critical path.

The inverse is also true, and it is why heterogeneous architecture is not a blanket recommendation. If profiling shows one stage owns nearly all the latency, a focused single-target port of that one stage is the right answer — and dragging the rest of the path onto a new target with it just adds integration surface for no gain. Heterogeneous mapping is a tool for paths with distributed bottlenecks; it is overkill for paths with a single dominant one.

There is a genuine cost on the other side of the ledger. Every additional target a path spans adds a deployment surface, a validation surface, and a boundary where data has to be marshalled from one runtime into another. A path that lives entirely in one CUDA process is simpler to ship and monitor than one that hands tensors between a CUDA kernel, a native C++ module, and a WASM sandbox. That marshalling cost — serialising at the boundary, keeping the two build toolchains in sync, testing each target’s numerical behaviour — is real and it eats into the latency gain. The break-even is straightforward to reason about once you have the numbers.

Worked example: is the split worth it?

Assume a path profiled at 100 ms end to end, illustrative figures to show the arithmetic:

Model forward pass: 60 ms, GPU-bound, currently on CPU. Moving to GPU/CUDA is projected to bring it to ~15 ms.
Tokeniser + post-processing: 30 ms, Python-overhead-bound. Porting to native C++ is projected to bring it to ~8 ms.
IO: 10 ms, unchanged (not compute-bound).

Uniform C++ port: fixes the 30 ms stage to ~8 ms, but the 60 ms model stage on CPU C++ maybe reaches ~45 ms. New total ≈ 63 ms.

Heterogeneous split: model → GPU (~15 ms), tokeniser → C++ (~8 ms), IO (10 ms). New total ≈ 33 ms — but add, say, ~4 ms of cross-target marshalling. Effective ≈ 37 ms.

The split reaches roughly 37 ms against the uniform port’s 63 ms. The measurable outcome is the avoided cost of porting the whole path to one target when only some stages needed a different one, plus the quantified gain from routing each stage to its correct target. These are illustrative numbers; the point is that the marshalling overhead has to be subtracted from the split’s gain, and the decision only holds if the remaining gain still clears your latency target.

When is keeping a stage in Python still the right call?

Not every stage earns a port. If a stage contributes a small, non-critical slice of latency and is not on the path’s bottleneck, keeping it in Python — or wrapping just the hot loop in Cython, as our note on what “computationally expensive” really means in an inference path explains — can hit the latency target with far less engineering and far less integration risk. The software porting decision more broadly is always a cost-benefit judgement, and “leave it in Python” is a legitimate outcome for stages that aren’t in the way.

This is where the discipline pays for itself. A heterogeneous mapping is not a mandate to port every stage; it is a mapping that tells you which stages to leave alone. The cheapest engineering is the port you correctly decided not to do.

The profiling and target-assignment work described here is the target-assignment step in our [inference cost-cut engagement](Inference Cost-Cut Pack), which follows the port-or-don’t-port decision and maps profiled stages onto heterogeneous compute targets. The broader engineering context for these targets lives on our GPU engineering practice page. The methodology that governs how per-stage targets are profiled and assigned before any mapping is committed is part of our porting and performance-assessment approach; release-readiness becomes its own concern once a path spans multiple targets, because each target adds its own deployment and validation surface.

FAQ

How does heterogeneous architecture work?

It treats an inference path as a system of distinct stages — model compute, pre/post-processing, and IO — rather than one block, and lets each stage run on the compute target that best removes its bottleneck. In practice that means the forward pass might run on GPU/CUDA, the tokeniser on native C++, and a client-side path on WASM, all within one logical path, matched to where profiling says the cost lives.

How do we decide which compute target each stage of an inference path should run on?

You attribute the latency per stage first, then assign each stage to the target that moves its specific bottleneck. GPU/CUDA for data-parallel model compute, native C++ for Python-overhead-bound branchy work, WASM/Pyodide when the constraint is client-side or no-backend deployment. IO-bound stages are fixed by batching or layout work, not by any compute-target change.

When does splitting an inference path across multiple targets beat porting the whole path to one?

Splitting wins when the profiled bottlenecks live in different stages that want different targets, so a single-target port would fix the wrong stage. If one stage owns nearly all the latency, a focused single-target port of just that stage is better, and dragging the rest along only adds integration surface for no gain.

How does profiling per-stage bottleneck attribution feed the target-assignment decision?

Profiling splits total latency into named contributions — model compute versus Python overhead versus IO — so each stage’s dominant cost is identified before any target is chosen. The mapping rule then assigns each stage to the target that removes that specific cost, which is why the attribution must come before the assignment, not after.

What integration and maintenance cost does a heterogeneous path carry versus a uniform port?

Every additional target adds a deployment surface, a validation surface, and a boundary where tensors must be marshalled between runtimes, plus the need to keep separate build toolchains and numerical behaviour in sync. That marshalling overhead must be subtracted from the split’s latency gain, and the split only holds if the remaining gain still clears the target.

When does keeping a stage in Python (or Cython) alongside a native/WASM stage still hit the latency target?

When the stage contributes a small, non-critical slice of latency and is not on the path’s bottleneck, keeping it in Python or wrapping just the hot loop in Cython can meet the target with far less engineering and integration risk. The cheapest engineering is the port you correctly decided not to do.

Where teams get burned is treating the language rewrite as the decision and discovering, a quarter later, that the bottleneck never lived where the port went. The failure class is uniform porting against distributed bottlenecks — and the artifact that prevents it is a per-stage target assignment grounded in profiling, not in a preference for one runtime.