Heterogeneous System Architecture: How CPU/GPU/WASM Targets Divide Inference Work

A heterogeneous system is not defined by the compute units your code assumes exist. It is defined by the ones your deployment target actually reaches at runtime. That distinction is where most browser and WASM ports quietly break.

Here is the pattern we see regularly. A team builds an inference path on a native box — CUDA offload for the compute-bound matmuls, a threaded pipeline feeding the GPU, shared memory-mapped buffers between stages. It runs well. Then someone asks for an in-browser demo, or a privacy-sensitive deployment that has to run client-side, and the plan becomes “compile it to WASM with Pyodide.” The assumption baked into that plan is that a heterogeneous architecture is a property of the code. It is not. It is a property of the target.

What does heterogeneous system architecture actually mean here?

At its most literal, a heterogeneous system is one with more than one kind of processing element — a CPU alongside a GPU, or an NPU, DSP, or FPGA — connected by some memory fabric, where work is placed on whichever unit executes it best. The intuition most engineers carry is the workstation model: a CPU that orchestrates and hands compute-bound kernels to a discrete GPU over PCIe, with the runtime (CUDA, cuDNN, NCCL for multi-device) managing placement and transfer.

That model is correct on a native box. The mistake is treating it as portable. The heterogeneity you can exploit is exactly the set of execution units reachable from your process at runtime — and that set changes with the target, not with the source. A well-written CUDA path exposes nothing extra to a runtime that has no CUDA. The broader treatment of heterogeneous architecture across CPU, GPU, and WASM targets works through the same point from the deployment angle; this article stays on the mapping problem — how you enumerate what a target genuinely offers before you commit to it.

So the first citable claim is blunt: a heterogeneous system is defined by which compute units and memory domains are reachable at runtime, not by the hardware physically present in the machine. A laptop with a capable discrete GPU can present an almost-uniprocessor execution surface to code running inside a browser tab. The silicon is there. Your process cannot reach it the way native code could.

Which units does a WASM/browser target actually expose?

The gap between “hardware present” and “hardware reachable” is widest in the browser sandbox. It helps to lay the two targets side by side.

Reachable execution units by target

Execution capability	Native CPU/GPU box	Browser / Pyodide WASM
CPU cores, general compute	All cores, direct threading	Main thread guaranteed; extra cores only via Web Workers
SIMD	Full AVX/AVX-512 via `-march`	WASM SIMD (128-bit fixed width), if the build enables it
Discrete GPU (CUDA)	Yes, direct	No — no CUDA, no native driver access
GPU compute (portable)	CUDA / OpenCL / Vulkan	WebGPU compute shaders, where the browser supports it
Shared-memory threading	pthreads, memory-mapped buffers	Only with `SharedArrayBuffer` + cross-origin isolation headers
Memory model	Multiple domains, DMA across PCIe	Single linear WASM heap per module
Native accelerators (NPU/DSP)	Vendor SDK, if drivers loaded	Not reachable from the sandbox

Read that table as a subtraction. Almost every offload lever a native heterogeneous design leans on — direct CUDA, unrestricted pthreads, DMA between memory domains — is either gone or gated behind a browser feature that requires deliberate build and deployment configuration. The single most consequential row is the memory model. Native heterogeneous code assumes distinct memory domains it can move data between; a WASM module runs against one linear heap, so the “shared buffer between a CPU stage and a GPU stage” pattern has no direct analogue unless WebGPU is in play.

This is why the collapse is structural rather than a tuning problem. In our experience porting compute-bound paths, teams expect a performance penalty and instead hit an availability wall: the offload target the design assumed simply is not addressable. That is claim two — the browser sandbox removes direct CUDA, constrains threading to Web Workers, and presents a single linear memory model, which eliminates most native heterogeneous offload paths rather than merely slowing them.

Why does the sandbox collapse offload instead of just taxing it?

The sandbox exists to isolate untrusted code from the host. That isolation is the whole point, and it is also why offload disappears. Direct GPU driver access would breach the sandbox, so there is no CUDA and no native OpenCL from a page. Unrestricted native threads with shared memory are a security and stability hazard in a browser, so concurrency is mediated through Web Workers — separate execution contexts that, absent SharedArrayBuffer and the cross-origin isolation headers, communicate by copying messages rather than sharing memory. Even when you enable shared memory, you have bought back threading, not the multi-domain memory model.

The practical consequence: under a straight Pyodide/WASM port with no additional acceleration, a compute-bound inference workload runs largely on the main thread. Whatever the native design offloaded to the GPU now executes as interpreted-or-compiled WASM on one core, competing with the UI event loop. The compiler-flag choices that shape Pyodide performance — enabling WASM SIMD, threading support, optimization level — determine how much that single-thread path can claw back, but they operate inside the ceiling the sandbox sets. They tune the reachable units; they do not add unreachable ones.

When does WebGPU restore heterogeneous offload, and where does it stop?

WebGPU is the one lever that puts a real second compute unit back on the table in the browser. It exposes GPU compute shaders through a portable API, so a matmul-heavy kernel that would have gone to CUDA on the native box can instead run as a WebGPU compute pass. Our walkthrough of the WebGPU specification as a portable GPU compute API covers the programming model; the relevant point for mapping is what it restores and what it does not.

WebGPU restores a GPU execution unit and a device-local memory domain distinct from the WASM heap — genuine heterogeneity. What it does not restore is the native-CUDA feature surface. Availability is per-browser and per-device (it degrades or is absent on older hardware and some configurations). Kernels must be rewritten in WGSL rather than reused from CUDA. Data crosses the JS/WASM boundary to reach the GPU, so the transfer cost that PCIe imposed on the native box has a browser analogue you must budget for. So claim three: WebGPU re-introduces a reachable GPU compute unit and a separate memory domain for in-browser inference, but only where the browser and device support it, and only for kernels rewritten to its API — it is not a drop-in for a native CUDA path.

The honest framing is that WebGPU turns a uniprocessor-looking target back into a two-unit heterogeneous one, conditionally. You plan for the condition, or you plan for the fallback.

How do you inventory the reachable execution units before a port?

The point of the mapping is to convert “we assume we can offload” into a documented answer before anyone writes port code. Treat it as an inventory step, not a vibe check. This is the same execution-unit map that feeds a broader profiling baseline — pairing it with profiling the Python inference path before a C++ or WASM port tells you both what can be offloaded and what would benefit from it.

Execution-unit inventory checklist

Assumptions for this checklist: the workload is a compute-bound inference path with a defined latency budget, and the candidate target is a browser/Pyodide deployment.

Enumerate CPU concurrency. Is the target single-thread-only, or can you use Web Workers? Do you control the response headers needed for SharedArrayBuffer and cross-origin isolation?
Confirm SIMD. Does the toolchain emit WASM SIMD, and does the target browser execute it? A 128-bit fixed width is the ceiling — size expectations to it, not to AVX-512.
Test WebGPU availability. On the actual target devices and browsers, does a compute-shader path initialise? Record the failure fraction, not just the happy path.
Map the memory model. Which stages assumed shared or memory-mapped buffers? Each one needs a single-heap or JS/WASM-boundary redesign.
Rule out native accelerators. Any NPU/DSP/FPGA the native design used is off the table in the sandbox — mark those kernels as CPU-or-WebGPU-only.
Measure the residual main-thread share. With the above settled, what fraction of the workload has nowhere to go but the main thread, and does that fit the latency budget?

The output is concrete: a per-target list of reachable compute units, the share of work that lands on the main thread, and a go/no-go on whether the target’s heterogeneity supports the budget. That inventory is what lets the [Pyodide fit assessment in our inference-cost-cut work](Inference Cost-Cut Pack) compare a WASM path honestly against a native C++/CUDA one — the same profiling baseline scoped to the units the target can actually reach. If you want the starting point rather than the port decision, the inference cost work overview frames where this sits.

There is a release-readiness dimension too. A path that assumes offload the sandbox does not provide will fail readiness checks late, when the demo is due. Treating the reachable-execution-unit set as a gate — the way an AI infrastructure readiness process treats target capability — moves that failure from launch week to planning week.

FAQ

How does heterogeneous system architecture work in practice?

A heterogeneous system places work across more than one kind of processing element — typically a CPU orchestrating and a GPU or accelerator handling compute-bound kernels — connected by a memory fabric. In practice the exploitable heterogeneity is the set of compute units and memory domains your process can actually reach at runtime, which is a property of the deployment target, not of the source code.

Which compute units and memory domains does a WASM/browser target actually expose compared to a native CPU/GPU box?

A native box exposes all CPU cores with direct threading, a discrete GPU via CUDA/OpenCL, multiple memory domains with DMA, and any loaded native accelerators. A browser/Pyodide target guarantees the main thread, offers extra cores only through Web Workers, has no CUDA, presents a single linear WASM heap, and can reach a GPU only through WebGPU where supported. Most native offload levers are removed rather than slowed.

Why does the browser sandbox collapse most native heterogeneous offload — no CUDA, limited threading, single memory model?

The sandbox isolates untrusted code from the host, and that isolation is exactly what removes offload. Direct GPU driver access would breach the sandbox, so there is no CUDA; unrestricted shared-memory threads are a hazard, so concurrency is mediated through Web Workers; and a module runs against one linear heap, so the multi-domain memory pattern has no direct analogue unless WebGPU is used.

How do you inventory the reachable execution units for a deployment target before committing to a port?

Run an execution-unit inventory: enumerate CPU concurrency and whether Web Workers and SharedArrayBuffer are available, confirm WASM SIMD emission and support, test WebGPU initialisation on the real target devices, map which stages assumed shared memory, rule out native accelerators, and measure the fraction of work that lands on the main thread. The output is a per-target unit list and a go/no-go against the latency budget.

When does WebGPU restore some heterogeneous offload for in-browser inference, and what are its limits?

WebGPU restores a reachable GPU compute unit and a device-local memory domain by exposing compute shaders through a portable API. Its limits: availability is per-browser and per-device, kernels must be rewritten in WGSL rather than reused from CUDA, and data crosses the JS/WASM boundary with a transfer cost you must budget. It is a conditional second compute unit, not a drop-in CUDA replacement.

How much of a compute-bound inference workload realistically stays on the main thread under Pyodide/WASM?

Under a straight Pyodide/WASM port with no additional acceleration, a compute-bound path runs largely on the main thread, since the work the native design offloaded to the GPU now executes as WASM on one core. Compiler flags — WASM SIMD, threading, optimization level — and Web Workers can reduce that share, but only WebGPU or offloading to workers actually moves work off the main thread.

How does the heterogeneity of the target feed the Pyodide-versus-native port decision?

The reachable-execution-unit inventory quantifies how much of the workload can be offloaded on the target versus how much collapses onto the interpreter, and pairing it with a profiling baseline shows both what can move and what would benefit from moving. That evidence lets a Pyodide path be compared honestly against a native C++/CUDA one against the same latency budget, turning “can we offload this?” into a documented go/no-go.

The question worth carrying out of any port planning session is not “how do we make this fast in the browser” but “what does this target genuinely let us reach” — because heterogeneity is a deployment-target decision, and a path that assumes offload the sandbox never provides fails the release-readiness gate every time. Map the units first; the port decision follows from the map.