A team searching “heterogeneous systems architecture” usually arrives with one assumption baked in: point the runtime at a machine with a CPU, a GPU, and maybe an accelerator, and the toolchain will quietly route each piece of work to whatever unit runs it fastest. Add a WASM Python target for the browser edge, and the mental model gets even more optimistic — heterogeneity as free parallelism, boundaries hidden, everything faster by default. That is not what heterogeneous architecture buys you. It buys you targeted placement — the ability to put the right work on the right compute unit. What it does not buy you is a free lunch across the boundaries between those units. Every time data crosses from CPU memory to GPU memory, or from a native path into a sandboxed interpreter, there is a marshalling cost that survives the move. Heterogeneity gives you the map; it does not walk the road for you. The practical consequence is simple and expensive to ignore. If you distribute an inference path across CPU, GPU, and accelerator without first knowing which unit actually owns the bottleneck, you can end up with a costlier, more complex version of the same latency you started with — now spread across more hardware and more copies. The whole discipline of heterogeneous placement comes down to a question you have to answer before you split anything: where does the time really go? How does heterogeneous systems architecture work in practice? A heterogeneous system is a machine with more than one kind of compute unit sharing a workload — most commonly a general-purpose CPU alongside a GPU, and increasingly a dedicated accelerator (an NPU, an AI engine, an FPGA image) or a sandboxed runtime like WebAssembly. Each unit has a different execution model, a different memory it treats as “local,” and a different cost to reach anything outside that local memory. The word that matters is boundary. On a discrete GPU, the device has its own high-bandwidth memory (HBM), and the CPU has system DRAM; the two are joined over PCIe or, on higher-end platforms, NVLink. On the CPU side, an interpreter like CPython owns a process address space and executes bytecode. In a WASM target, a sandbox owns a linear memory region that native code cannot directly touch. Heterogeneity is not “many processors” — it is many memory owners, joined by boundaries with a price. The runtime does not transparently dissolve those boundaries. Frameworks like PyTorch and TensorRT make the copies convenient — tensor.to("cuda") is one line — but the copy still happens, still consumes PCIe bandwidth, and still shows up in a profile. The architecture defines who owns what and where the interpreter or sandbox sits in the map. Deciding what to place where is an engineering decision, not a runtime automatism. Our own heterogeneous architecture for inference walkthrough works through the CPU/GPU/WASM split with concrete placement examples; this article is the conceptual frame beneath it. What compute units make up a heterogeneous system, and who owns memory? It helps to lay the units out with their memory ownership stated plainly, because that ownership is what determines the cost of every cross-unit hop. Compute unit What it owns Local memory Reaches other units via Typical role in inference CPU (host) Process address space, orchestration, control flow System DRAM PCIe / NVLink to GPU; direct call into WASM sandbox Pre/post-processing, tokenisation, glue, small ops GPU (device) Massively parallel kernels HBM / GDDR PCIe / NVLink back to host Dense matmul, attention, convolution Accelerator (NPU / AI engine / FPGA) Fixed-function or dataflow compute On-chip SRAM / tiled memory Vendor interconnect to host Quantised layers, specific operators WASM runtime Sandboxed linear memory Isolated linear memory Host calls in/out of the sandbox Portable browser-edge inference, no native GPU The table makes the core rule extractable: in a heterogeneous system, no single unit owns all the memory — and the cost of an inference path is dominated by who has to hand data to whom, and how often (observed pattern across porting engagements; not a single benchmarked figure). The CPU is almost always the orchestrator. It holds the control flow, decides what to dispatch, and marshals inputs and outputs. That role is easy to underestimate. A path that looks GPU-bound on paper can spend the majority of its wall-clock time on the host — copying tensors, running the Python interpreter, or waiting on a memory transfer — which is exactly the failure the naive “the runtime will route it” model produces. If you want to see where that host time actually accumulates, what “computationally expensive” means in an inference path traces the cost class by class. Where does a WASM Python runtime sit inside the architecture? This is the placement that trips people up most, because a WASM Python path — for example, running through Pyodide in a browser — feels like a shortcut. Ship the same Python inference code, run it client-side, skip the server GPU. In reality it is one specific placement inside the heterogeneous map, with its own ownership and its own offload story. A WASM runtime owns an isolated linear-memory region. It is a CPU-class execution target: no direct access to a discrete GPU’s HBM, no CUDA, and — for a Python-in-WASM stack — an interpreter running inside the sandbox on top of a compiled-to-WASM CPython. What it owns is portable execution and isolation. What it offloads is limited: WebGPU can hand parallel work to the browser’s GPU, but crossing from WASM linear memory into a WebGPU buffer is another marshalling boundary, not a free bridge. Our WebGPU spec explainer covers what that browser-side compute path can and cannot reach. So a WASM Python placement is a genuine option in the architecture — it is just not a way around the boundaries. It moves the workload onto a CPU-class, sandboxed unit and inherits every cost that CPU-class inference already had, plus the sandbox’s own overhead, plus a cold-start penalty on first load. Whether that is the right placement depends entirely on where the workload’s real cost lives, which is the same question every other placement decision comes down to. What marshalling cost does crossing a boundary add, and where does it bite? Marshalling is the work of getting data from one unit’s memory into a form and location the next unit can use: copies, layout conversions, serialisation into and out of a sandbox, synchronisation barriers. It is not overhead you can optimise away with a compiler flag — it is intrinsic to the boundary existing. Where it bites for inference: Small, frequent transfers are worse than a few large ones. A path that shuttles a tensor to the GPU, runs one small kernel, pulls it back, does a little CPU work, then pushes it again pays the PCIe latency tax on every round trip. The device is often idle waiting for data. Interpreter-boundary cost on a CPU/Python path — bytecode dispatch, object marshalling into native extensions — can dominate small-op-heavy models even when the heavy math is fast. Sandbox crossings in a WASM path add serialisation each time control leaves the linear-memory region, and a cold start that a warm native process never pays. The reason this matters to a placement decision: marshalling cost survives the move. If you split a path to put dense matmul on the GPU but the workload is dominated by many small host-side operations, you have added boundary crossings without relieving the thing that was actually slow. The relevant public physics is unforgiving here — PCIe bandwidth and latency are fixed by the platform, and per NVIDIA’s published platform specifications the gap between an NVLink-connected topology and a PCIe one is exactly the kind of boundary cost that determines whether a split pays off. PCIe topology and NUMA effects on GPU data movement get into why the host side of that boundary is more than a single number. When does distributing an inference path earn its cost? Here is the decision, framed as a rubric rather than a rule. A cross-unit split earns its cost when the work you move to a faster unit is large enough that the compute saving on that unit clears the marshalling cost of getting the data there and back. Score a candidate split against these: Is the bottleneck on the unit you are moving work off? If the profile says 80% of wall-clock time is host-side interpreter overhead, moving matmul to the GPU changes little. Distribute only what is slow. Is the moved work coarse-grained? One large kernel amortises a transfer; a thousand tiny kernels do not. Batch and fuse before you split. Does the boundary have headroom? An NVLink or on-package interconnect tolerates far more traffic than PCIe. Know your topology. Is the receiving unit actually a fit for the operator? An accelerator optimised for quantised convolution may be slower than a GPU for a general attention kernel. Placement is per-operator, not per-model. If a candidate split scores well on all four, distribute. If it fails on the first — the bottleneck is not on the unit you are relieving — stop, because no amount of clever placement fixes a workload whose cost lives on a single unit that heterogeneity does not touch. Keeping it on one well-tuned unit is frequently the faster and cheaper answer. A grounded read on this feeds directly into the runtime-fit evaluation inside our Inference Cost-Cut Pack, where a defensible unit-fit verdict is exactly what a port decision needs before engineering is funded. How do we profile a heterogeneous path before committing to a port? You cannot reason about placement from architecture diagrams alone. You measure. The goal is a profiled attribution of time spent on each compute unit: how much on the CPU interpreter, how much in GPU compute, and — the number people forget — how much in cross-boundary marshalling. Concretely, that means profiling the current path before touching the topology: instrument the Python layer to separate interpreter time from native time, use the GPU vendor’s profiler (Nsight for CUDA paths) to separate kernel execution from memory-copy time, and account explicitly for host-device transfer as its own line item rather than folding it into “GPU time.” When a copy shows up as a large fraction of the total, the split you were about to fund would have made it worse. We treat this profiling baseline as the entry gate to any heterogeneous decision — profiling the Python inference path before a C++ or WASM port lays out the mechanics we use, and it is the same discipline the porting and performance-assessment methodology applies before a heterogeneous split is funded. How do we tell whether heterogeneity relieved the bottleneck or just relocated it? This is the acceptance test, and it is different from “is it faster.” A split can shave a few milliseconds off one unit while adding marshalling cost that eats most of the gain — technically faster, economically a wash, and now more fragile to maintain across mixed runtimes. The honest check is a before/after attribution profile: run the same representative load on the single-unit path and on the distributed path, and compare where the time goes, not just the total. If the bottleneck’s line item shrank on its original unit and did not simply reappear as transfer time on a boundary, heterogeneity relieved it. If the total moved a little but a new marshalling line item swallowed most of the saving, you relocated the bottleneck rather than removing it — and you now own more complexity for less benefit. Release-readiness for a mixed-runtime deployment adds the cross-unit boundary, memory-ownership, and cold-start checks that this test does not cover on its own. FAQ How should you think about heterogeneous systems architecture in practice? A heterogeneous system runs one workload across more than one kind of compute unit — typically CPU, GPU, and sometimes an accelerator or WASM runtime — each with its own local memory and execution model. In practice it means targeted placement of the right work on the right unit, joined by boundaries that cost real time to cross. The runtime does not transparently route work to whatever is fastest; deciding what goes where is an engineering decision. What compute units make up a heterogeneous system, and who owns memory across each boundary? The common units are a CPU that owns system DRAM and orchestration, a GPU that owns HBM and runs parallel kernels, an optional accelerator with its own on-chip memory, and a WASM runtime that owns an isolated linear-memory region. No single unit owns all the memory. The CPU is usually the orchestrator, marshalling data to and from every other unit — which is why a path that looks GPU-bound can spend most of its wall-clock time on the host. Where does a WASM Python runtime sit inside a heterogeneous architecture, and what does it own versus offload? A WASM Python path is a CPU-class, sandboxed placement inside the architecture, not a shortcut around it. It owns portable execution and isolation inside a linear-memory region; it has no direct access to a discrete GPU. It can offload parallel work to the browser GPU via WebGPU, but crossing from WASM linear memory into a WebGPU buffer is another marshalling boundary, plus a cold-start penalty on first load. What marshalling cost does moving data across compute-unit boundaries add, and where does it bite for inference? Marshalling is the copying, layout conversion, serialisation, and synchronisation needed to hand data from one unit’s memory to the next; it is intrinsic to the boundary, not removable by a compiler flag. It bites hardest on small, frequent transfers that pay PCIe latency on every round trip, on interpreter-boundary cost for small-op-heavy CPU paths, and on sandbox crossings in WASM. The cost survives any split, so it can eat the gain from moving work to a faster unit. When does distributing an inference path across CPU/GPU/accelerator earn its cost versus keeping it on one unit? A split earns its cost when the work moved to a faster unit is large enough that the compute saving clears the marshalling cost of getting data there and back. Check four things: the bottleneck is on the unit you are relieving, the moved work is coarse-grained, the boundary has bandwidth headroom, and the receiving unit fits the operator. If the bottleneck lives on a single unit that heterogeneity does not touch, keeping it there is usually faster and cheaper. How do we profile a heterogeneous path to confirm the right work landed on the right unit before committing to a port? You measure before touching the topology, producing a time attribution across each unit: CPU interpreter time, GPU kernel time, and cross-boundary transfer time as its own line item. Separate interpreter time from native time in the Python layer, use the vendor GPU profiler to split kernel execution from memory copies, and account for host-device transfer explicitly. If a copy is a large fraction of the total, the split you were about to fund would have made it worse. How do we tell whether heterogeneity relieved the bottleneck or just relocated it? Run the same representative load on both the single-unit and distributed paths and compare where the time goes, not just the total. If the original bottleneck’s line item shrank and did not reappear as transfer time on a boundary, heterogeneity relieved it. If the total barely moved and a new marshalling line item swallowed most of the saving, you relocated the bottleneck and now carry more complexity for less benefit. The map is worth having. But a heterogeneous architecture answers “where could this work run” — it never answers “where should it.” That second question belongs to a profile, and the failure worth naming is the one where a team distributes a path across three units before ever confirming which one actually owned the wait.
