HSA Programming Explained: Shared-Memory GPU Compute and What It Means for Inference Cost

A serving config wins a kernel-only benchmark, then loses in production. The kernel was never the whole story. HSA programming is the missing frame: it forces you to account for the memory-transfer and coherency path a request pays for, not just the compute peak once data has landed on the device.

Heterogeneous System Architecture (HSA) sounds like a low-level GPU concern — a topic for driver authors and runtime engineers, not for the person who owns the serving bill. That reading is what makes it dangerous. HSA is fundamentally a memory-model idea, and memory movement is where a surprising share of your cost-per-request hides. If you only measure throughput after the tensors have landed on-device, you are measuring the fastest part of the request and ignoring the part that most often decides whether a config is actually cheaper.

What should you know about HSA programming in practice?

HSA is a specification for how a CPU and one or more accelerators share a coherent view of memory. The two ideas that matter for cost are unified virtual memory — a single address space the host and device both understand, so a pointer means the same thing on both sides — and cache-coherent shared allocations, where a region written by one processor becomes visible to the other without an explicit copy across the bus. In the classic, non-HSA model you allocate host memory, allocate device memory, and stage data between them: cudaMemcpy-style transfers over PCIe, a kernel launch, then a copy back. HSA collapses that staging by letting the accelerator operate on memory the host also owns.

In practice you rarely program raw HSA in an inference stack. You encounter its ideas through the abstractions built on top of it — CUDA’s Unified Memory, ROCm’s HIP managed allocations, torch pinned and shared tensors, or the zero-copy paths inside a runtime like TensorRT. The reason to understand the underlying model is that these abstractions do not remove the transfer cost; they relocate and reshape it. A managed allocation can still trigger a page migration under the hood. Coherency is not free — it is a traffic pattern with its own latency. Knowing that is the difference between reading a benchmark and understanding it.

What problem does unified shared memory solve compared to separate address spaces?

Separate address spaces impose an explicit, visible tax on every request that crosses the boundary. Consider an inference path with a preprocessing step on the CPU — tokenisation, image resizing, feature assembly — followed by GPU compute. In the copy-based model, each request pays: allocate device buffer, copy input host→device, launch, copy output device→host, free. Under realistic concurrency those copies contend for the same PCIe link and the same DMA engines that the compute stream needs, so the transfer cost is not merely additive — it competes with the work you actually care about.

Unified shared memory attacks that boundary tax. When the model is right for the workload, the input the CPU produced is already visible to the GPU with no staging copy, and the coherency fabric handles visibility on demand. The honest version of the comparison is not “kernel A is faster than kernel B” — it is “request path A costs less than request path B once movement and synchronisation are counted.” That reframing is the entire point. It is also why a config that looks strong in isolation, on a device-only microbenchmark, can quietly lose its advantage the moment host-device copies re-enter the accounting.

This is the same discipline we apply when spec-ing the compute behind a production AI feature: the cost that decides the bill is the cost of the whole request, not the cost of the hottest kernel in it.

How do host-device copies and coherency overhead show up in per-request cost?

They show up in three places, and each one is measurable if you instrument for it rather than assuming it away.

First, in wall-clock latency at the request level. A copy that takes a few hundred microseconds looks trivial next to a multi-millisecond decode, but at high concurrency the copies serialise against each other on a shared link and inflate p95 more than the median. The tail is where serving SLAs live, so this is not a rounding error.

Second, in effective utilisation. When the compute stream stalls waiting for a transfer to complete, the accelerator is billed for time it spent idle. A kernel benchmark reports the busy interval; a request benchmark reports the interval you actually pay for, including the stalls. The gap between those two numbers is the overhead HSA-aware reasoning is trying to expose.

Third, in coherency traffic under the unified model. Unified memory is not a magic eraser. If the access pattern forces frequent page migrations back and forth between host and device — a “ping-pong” pattern where both processors keep touching the same pages — you can pay more than an explicit, batched copy would have cost, because migration granularity and fault handling carry overhead the profiler will happily show you. The point is not that shared memory is always cheaper. The point is that you should measure which regime you are in.

To measure any of this you need the transfer and synchronisation timeline, which is exactly what GPU profiling supplies when a runtime or compiler reshapes the memory path. Attribution — how much of the request went to movement versus compute — is only possible with that timeline in hand.

When does an HSA shared-memory model actually reduce cost-per-request?

Not always, and the “not always” is the useful part. The shared-memory model earns its keep under specific conditions, and it is a liability under others. The table below is a decision rubric, not a ranking — read it against your own serving path.

Decision rubric: shared-memory path vs explicit-copy path

Condition in your serving path	Shared-memory (HSA-style) tends to win	Explicit-copy / staged tends to win
Data touched once per request, streamed	Yes — no round-trip staging needed	No
Frequent CPU↔GPU hand-off within one request	Yes — coherency avoids repeated copies	Only if hand-offs are large and batchable
Same pages re-written by both sides repeatedly	No — migration ping-pong dominates	Yes — one batched copy beats many faults
Small, latency-sensitive inputs at high concurrency	Often — removes per-request copy contention	Rarely
Large, predictable batched transfers	Sometimes	Yes — DMA on a dedicated stream overlaps well
Hardware with true coherent fabric (APU / NVLink-C2C class)	Strongly	—
Discrete accelerator over PCIe only	Weakly — “unified” still migrates over the bus	Often

Evidence class: observed-pattern, drawn from serving-path profiling in TechnoLynx engagements; the direction generalises but the crossover point is hardware- and workload-specific — measure it, do not assume it.

The single most important row is the last one. On hardware with a genuine coherent interconnect — an integrated APU, or an NVLink-C2C class link between CPU and GPU — unified memory can be close to free because there is no PCIe hop to migrate across. On a discrete card connected only by PCIe, “unified memory” is a programming convenience that still moves bytes over the same bus at the same bandwidth ceiling. The abstraction is identical; the physics is not.

How do you benchmark a serving path so transfer overhead is attributed separately from compute?

Build the benchmark to answer one question: of the per-request cost, how much is movement and how much is compute? A kernel-only microbenchmark cannot answer that by construction, because it starts the clock after the data has arrived.

Use this checklist to keep the accounting honest:

Measure end-to-end request latency, not kernel time. The clock starts when the request’s data is in the state it arrives in (host memory, typically) and stops when the response is ready to return.
Run at the concurrency you actually serve. Copy overhead that is invisible at concurrency 1 becomes the dominant tail contributor at concurrency 32 because transfers contend for the link.
Separate the streams and read the timeline. With CUDA streams (or the ROCm equivalent) on distinct queues, a profiler timeline shows transfer intervals and compute intervals as separate lanes — that separation is the attribution.
Report p95, not just the mean. Movement overhead concentrates in the tail; a mean hides it.
Test both memory models on the same hardware. Run the explicit-copy path and the shared-memory path against the identical workload so the difference is attributable to the memory model, not to some other variable.
Include allocation and free cost. Per-request allocation churn is real overhead; pooled or pinned buffers change the answer.

This is the same fairness principle that governs any config comparison: hold everything constant except the variable under test, and structure the runs so the number means what you think it means. We treat that structuring the way a benchmark spec organises serving configs — the comparison is only fair if the run conditions are pinned.

What does an HSA-aware versus copy-based comparison look like for a typical request?

Take a worked example, with the assumptions stated so you can adjust them to your own numbers.

Assumptions (illustrative). A request carries a ~4 MB input tensor to the GPU and returns a ~1 MB output. Compute is ~3 ms of kernel time. PCIe Gen4 x16 delivers on the order of 25 GB/s of usable host-device bandwidth per the interface’s published specification — so ~4 MB in is roughly 160 µs and ~1 MB out roughly 40 µs, call it ~0.2 ms of raw transfer at concurrency 1.

At concurrency 1, that 0.2 ms of copy sits next to 3 ms of compute — around 6% of the request. Easy to dismiss. Now raise concurrency to 32. The compute streams overlap and the accelerator stays busy, but the 32 concurrent transfers contend for one PCIe link with a fixed bandwidth ceiling. Copies that were 0.2 ms each now queue, and the tail request waits behind the ones ahead of it. In configurations we have profiled, this is where the copy-based path’s p95 diverges sharply from its median while the compute time barely moves — the overhead was always there, concurrency just made it visible.

The HSA-aware path, on hardware with a coherent interconnect, removes the staging entirely for the streamed-once pattern: the input the CPU produced is already visible to the GPU, so the per-request copy contention disappears and p95 tracks compute far more closely. On a discrete PCIe card, the same “unified” code still migrates those bytes over the same link — so the benchmark, if you built it correctly, will tell you the memory model did not help here and the honest move is to batch the copies instead.

Evidence class: the transfer arithmetic is derived from published PCIe Gen4 x16 specifications; the concurrency-divergence pattern is observed-pattern from serving-path profiling, not a single named benchmark result.

That is the whole lesson in one example. The memory model changed the request economics in one hardware regime and did nothing in another, and only a whole-path benchmark could tell the two apart.

How does HSA relate to CUDA unified memory and other vendor shared-memory approaches?

HSA is the vendor-neutral specification of the idea; the shipping implementations are the vendor expressions of it. NVIDIA’s CUDA Unified Memory gives you a single pointer valid on host and device with driver-managed migration and, on Grace-Hopper class hardware, hardware coherence over NVLink-C2C. AMD’s ROCm/HIP exposes managed and coherent allocations that trace directly to HSA’s lineage — HSA originated in the heterogeneous-computing work AMD drove. Intel’s oneAPI/Level Zero offers unified shared memory (USM) as a first-class allocation type. The abstractions differ in API surface and in how aggressively they migrate, but they answer the same question: can the accelerator operate on memory the host also owns, without an explicit staged copy?

For an inference stack that cares about portability across these vendors, the shared-memory model interacts directly with backend choice — the reasoning is close to what we cover for portable GPU inference backends on Linux, where the same code has to behave predictably across different memory subsystems. The practical takeaway is that “unified memory” is a portable programming concept whose cost is not portable at all. It depends on whether the underlying link is coherent silicon or a PCIe hop wearing a convenient API.

FAQ

What does working with hsa programming involve in practice?

HSA specifies how a CPU and accelerators share a coherent memory view through unified virtual memory (a single address space) and cache-coherent shared allocations. In practice you meet it through CUDA Unified Memory, ROCm/HIP managed allocations, or oneAPI USM rather than raw HSA — and those abstractions relocate transfer cost rather than removing it.

What problem does HSA’s unified shared memory solve compared to separate CPU and GPU address spaces?

Separate address spaces tax every request that crosses the CPU–GPU boundary with explicit staged copies that contend for the bus under concurrency. Unified shared memory lets the accelerator operate on memory the host also owns, removing the staging copy — but only when the access pattern and the hardware suit it. The honest comparison is request-path cost, not kernel speed.

How do host-device memory copies and coherency overhead show up in per-request inference cost?

They surface as inflated p95 latency when copies serialise on a shared PCIe link at concurrency, as billed idle time when the compute stream stalls waiting on a transfer, and as migration overhead when a “ping-pong” access pattern forces repeated page moves under unified memory. Each is measurable with a profiler timeline that separates transfer intervals from compute.

When does an HSA shared-memory model actually reduce cost-per-request, and when does it not matter?

It helps most when data is streamed once, when CPU↔GPU hand-offs are frequent within a request, when inputs are small and latency-sensitive at high concurrency, and — decisively — when the hardware has a truly coherent interconnect like an APU or NVLink-C2C link. It helps little on a discrete PCIe card, where “unified” memory still migrates bytes over the same bus, and it can hurt when both processors repeatedly rewrite the same pages.

How do you benchmark a serving path so memory-transfer overhead is attributed separately from compute?

Measure end-to-end request latency rather than kernel time, run at production concurrency, and place transfers and compute on separate streams so a profiler timeline attributes each. Report p95 (not just the mean), test both memory models on identical hardware and workload, and include allocation/free cost.

What does an HSA-aware versus copy-based comparison look like for a typical inference request?

At concurrency 1 a ~0.2 ms transfer next to ~3 ms of compute looks negligible, but at concurrency 32 those copies contend for one bandwidth-capped PCIe link and inflate the tail while compute barely moves. A coherent-interconnect HSA path removes that staging for streamed-once data so p95 tracks compute; a discrete PCIe card running the same “unified” code still migrates the bytes, and the benchmark tells you to batch copies instead.

How does HSA relate to CUDA unified memory and other vendor shared-memory approaches in practice?

HSA is the vendor-neutral specification; CUDA Unified Memory, ROCm/HIP managed allocations, and oneAPI USM are the shipping expressions of the same idea with different APIs and migration behaviour. The unified-memory concept is portable, but its cost is not — it depends entirely on whether the CPU–GPU link is coherent silicon or a PCIe hop behind a convenient API.

Where this leaves the cost question

The reason to learn HSA is not to write heterogeneous kernels; it is to stop trusting benchmarks that quietly amputate the movement half of a request. Once you account for the copy and coherency path, config selection reflects true per-request economics instead of a peak-throughput headline — and that attribution is precisely what an [inference cost-cut sprint](Inference Cost-Cut Pack) measures before and after, on the same serving path, for AI-infrastructure SaaS teams whose bill scales with every request.

The open question for your stack is narrower than “is shared memory faster.” It is: on your hardware, at your concurrency, does the memory model move the p95 you are billed for — and have you built a benchmark honest enough to tell you either way?