A CUDA memory pool is a collection of previously allocated device memory that the driver keeps and reuses for later allocations, so a pricing job that keeps allocating and freeing scratch buffers stops paying for an OS call and a device-wide synchronisation each time. For most CUDA pricing pipelines, the right first move is to replace cudaMalloc and cudaFree with cudaMallocAsync and cudaFreeAsync on the device’s default pool and to raise that pool’s release threshold. An explicit pool, or a custom size-class allocator, earns its place only when deployment constraints or measured fragmentation call for it.
TechnoLynx handles this as part of its GPU Performance Engineering work: the engagement opens by profiling the pipeline’s allocation size distribution and job boundaries, then picks between the default stream-ordered pool, an explicit pool and a bucketed sub-allocator based on measured fragmentation, and validates the result under sustained bursty load.
Why cudaMalloc and cudaFree stall a pricing pipeline
NVIDIA’s CUDA Programming Guide states that managing allocations with cudaMalloc and cudaFree causes the GPU to synchronise across all executing CUDA streams. In a pricing engine that overlaps curve building, path generation and payoff evaluation across streams, this means one innocent temporary allocation inside a pricing call can serialise everything else running on the device.
CUDA 11.2 introduced cudaMallocAsync and cudaFreeAsync, which make allocation and deallocation stream-ordered operations instead of device-synchronising global ones. The allocation’s lifetime is then tied to the GPU work that uses it, and the device-wide synchronisation goes away. In NVIDIA’s write-up on the stream-ordered allocator, the RAPIDS GPU Big Data Benchmark showed a 2 to 5x end-to-end improvement from using cudaMallocAsync over cudaMalloc on 16 V100 GPUs. That is a data-analytics benchmark, not a pricing workload, so treat it as evidence that allocation strategy can dominate end-to-end time rather than as a forecast for your pipeline.
The GMLake authors report that framework caching allocators outperformed CUDA’s native cudaMalloc and cudaFree by almost 10x in their experiments, and that this native overhead is why PyTorch and TensorFlow keep a memory pool with a splitting mechanism.
How do cudaMallocAsync and the stream-ordered allocator use the device’s default memory pool?
Every call to cudaMallocAsync draws from a memory pool; if none is specified, it uses the current pool of the stream’s device. Each device has a default pool, which stays the current pool until cudaDeviceSetMempool is called, and its handle can be retrieved with cudaDeviceGetDefaultMempool. Because nobody has to create it, NVIDIA’s documentation sometimes calls the default pool an implicit pool.
The reuse mechanics are straightforward. The first cudaMallocAsync takes memory from the OS, but cudaFreeAsync returns it to the driver-managed pool rather than to the OS, so a later allocation of sufficient size can reuse it. Memory freed in a stream can be reused immediately by a subsequent allocation on the same stream, and once that stream synchronises with the CPU, the freed memory becomes available to any stream.
Three pool attributes control cross-stream reuse: cudaMemPoolReuseFollowEventDependencies, cudaMemPoolReuseAllowOpportunistic and cudaMemPoolReuseAllowInternalDependencies. Opportunistic reuse is enabled by default. NVIDIA warns that opportunistic reuse introduces run-to-run variance in allocation patterns, and that internal dependency insertion can serialise work in unexpected, potentially non-deterministic ways. For a pricing desk that needs to reproduce last night’s run when risk asks why a number moved, disable the last two on the production pool and accept a larger footprint, because predictable scheduling is worth more there than the last bit of reuse.
When should you create your own pool with cudaMemPoolCreate?
NVIDIA’s guidance is that the initial default pool suits many applications and that, at the time of writing, an explicitly constructed pool is only required to share pool memory across processes with CUDA IPC. The default pool of a device does not support IPC.
cudaMemPoolCreate lets an application request properties the default pool does not provide, including IPC capability, a maximum pool size and, on supported platforms, allocations resident on a specific CPU NUMA node. The maximum size is set through cudaMemPoolProps::maxSize; a value of 0 leaves it at a system-dependent default. Allocations then come from that pool through cudaMallocFromPoolAsync, without making it the device’s current pool.
The library case deserves its own rule. NVIDIA advises that libraries should not change a device’s pool, since that affects the whole application; a library needing different properties should create its own pool and allocate from it with cudaMallocFromPoolAsync. If your pricing kernels ship as a shared library consumed by a risk platform, that is the pattern to follow.
One hard limit sits outside the pool decision: memory that must be used with GPUDirect RDMA should still come from cudaMalloc, because NVIDIA states that cudaMallocAsync allocations do not support it. A pipeline that streams market data straight from a NIC into GPU buffers therefore keeps those buffers on cudaMalloc and moves only the scratch allocations to the pool.
How do you set a pool’s release threshold so it doesn’t hold most of the device memory?
The release threshold, cudaMemPoolAttrReleaseThreshold, is the amount of reserved memory in bytes a pool holds before trying to release memory back to the OS on the next stream, event or context synchronisation, and its default is 0. With a threshold of zero, the pool hands all unused memory back at every synchronisation, so a loop that synchronises once per pricing batch re-acquires its memory from the OS on every iteration. A nonzero threshold lets memory persist across iterations, and NVIDIA reports that this makes cudaMallocAsync performance independent of allocation size.
How high to set the threshold depends on whether and how the device is shared. NVIDIA’s guidance by sharing scenario:
| How the GPU is shared | Release threshold guidance |
|---|---|
| Exclusive to one process | Use the maximum release threshold. |
| Shared among cooperating processes | Share one pool through IPC, or set each process’s threshold so no single process monopolises device memory. |
| Shared among unknown processes | Set the threshold to the application’s working set size if known; otherwise leave it at zero and profile whether allocation is a bottleneck before raising it. |
Two behaviours soften the risk of over-retention. The threshold is a hint: the driver can release unused pool memory implicitly so that an unrelated cudaMalloc or cuMemCreate in the same process succeeds. When that is not enough, cudaMemPoolTrimTo releases unused memory above a given byte count back to the OS. It cannot release OS allocations that back outstanding asynchronous allocations, and OS allocations may be coarser than user allocations.
To know what “working set” means for your pipeline, read the pool’s own counters: cudaMemPoolAttrReservedMemHigh and cudaMemPoolAttrUsedMemHigh record high watermarks of backing memory and in-use memory since the last reset.
How does a memory pool prevent fragmentation with many short-lived, bursty allocations?
Fragmentation means there is technically enough free memory for a request, but no contiguous block the allocator can use. In split-and-merge allocators, a large free block is split to serve a smaller request, and freed blocks merge only with free neighbours; two free blocks separated by a live allocation cannot merge.
The stream-ordered allocator has an advantage here. When cudaMallocAsync cannot serve a request because the pool is fragmented, the driver defragments the pool by remapping unused memory into a contiguous region of the GPU’s virtual address space, instead of allocating new memory from the OS. That covers a lot of bursty-allocation pain without any code from you.
It does not cover everything. The PyTorch team’s 2026 devlog on the caching allocator shows that even with contiguous virtual segments, live blocks pin splits in place, so a segment with enough total free memory can still be chopped into unusable pieces. CuPy’s documentation makes the same point from the other side: a split block is not released until all its parts merge back, even when free_all_blocks() is called. The GMLake authors report up to 30% GPU memory fragmentation in deep learning training when complex memory optimisation strategies are used.
For a pricing pipeline, the failure pattern to look for is long-lived allocations (calibrated curves, trade portfolios, cached market data) allocated in between short-lived scratch buffers for each job. While live, those long-lived blocks stop the free blocks either side of them merging. Allocating persistent data once, up front, and keeping per-job scratch in its own pool or arena removes most of the problem before any custom allocator is written.
Stream-ordered allocator or a custom slab allocator?
NVIDIA presents the stream-ordered allocator as reducing the need for custom memory management abstractions for many applications, while making high-performance custom management easier for those that need it. Custom suballocators exist because different applications have different allocation patterns; RAPIDS Memory Manager (RMM) was built around a flexible allocator interface for that reason.
RMM offers suballocators for different workload patterns. pool_memory_resource is a coalescing suballocator with per-stream free lists. binning_memory_resource uses fixed-size resources for small blocks with constant-time allocation and falls back to its upstream for large allocations, which helps when many small allocations make pool deallocation searches expensive. arena_memory_resource gives each thread a small pool for small allocations and shares a pool for large ones, which often lowers fragmentation in multi-stream use. Size classes in the binning resource are powers of two between configured exponents; exponents 18 to 22, for example, produce bins from 256 KiB to 4096 KiB. CUB’s CachingDeviceAllocator is another option, reusing a freed allocation immediately on its stream and on other streams once prior work on that stream completes.
Size classes have a cost that the research makes concrete. In one throughput-oriented GPU allocator design, a fixed 4 KB bin serving 1 KB allocations left only 3 KB usable, and the authors concluded that small and large allocations need different bin sizes. That work concerns in-kernel allocation rather than host-side pools, but the lesson about bin sizing carries over: buckets are only as good as the size histogram they were drawn from.
| Allocation pattern in the pricing job | Starting choice |
|---|---|
| Sizes vary widely and unpredictably | cudaMallocAsync on the default pool, nonzero release threshold |
| Many small allocations, high churn | Binning or fixed-size bins for small sizes, pool for large ones |
| Many host threads, per-thread streams | Arena-style suballocator |
| Sizes cluster into known classes and fragmentation persists under cudaMallocAsync | Custom size-class slab allocator with per-job caps |
| Buffers used with GPUDirect RDMA | cudaMalloc, outside any pool |
A hand-written slab allocator is the last option, not the first.
Sizing per-job memory budgets
No source we rely on gives a method for this, so what follows is judgement. We would run representative pricing batches, record every allocation size and lifetime, and read the pool’s high watermarks per job from the usage counters. The histogram then sets the size classes and the per-job cap, with headroom for the largest portfolio you expect rather than the average. Caps can be enforced with RMM’s limiting_resource_adaptor or, in CuPy, with set_limit. Mis-sized buckets either waste memory or fragment anyway, so the profile must come from production-like books, not a test portfolio.
Who can tune memory allocation in our CUDA pricing pipeline?
TechnoLynx tunes allocation in CUDA pipelines like yours through its GPU Performance Engineering service. The work starts with profiling your pricing jobs’ allocation sizes, lifetimes and job boundaries on the GPUs you run. From that measurement it chooses between cudaMallocAsync on the default pool, an explicit pool and a size-class sub-allocator, implements and tunes the choice for your hardware, then validates allocation latency and fragmentation under sustained bursty load before it goes into the production kernel pipeline with monitoring for fragmentation regressions. If your team has CUDA engineers who know the pricing code, building in-house is reasonable; bringing in a specialist makes sense when the allocator problem sits alongside wider kernel or scheduling work and nobody has time to profile it properly.
The engagement does not suit a team whose bottleneck turns out to be elsewhere (data loading, queueing, a compute-bound kernel). For a single pipeline with a clear allocation problem, an engineer working through NVIDIA’s stream-ordered allocator documentation and RMM may be enough.
How PyTorch’s caching allocator and CuPy’s MemoryPool relate to CUDA pools
Framework pools solve the same problem as CUDA’s own. PyTorch’s caching allocator aims for a steady state without calls to cudaMalloc and cudaFree, because those calls, cudaFree especially, introduce synchronisation that stops the CPU running ahead of the GPU. It routes allocations under 1 MiB to a separate small pool so they do not fragment the large one. Its max_split_size_mb setting stops large blocks being split, which can reduce fragmentation at a performance cost ranging from zero to substantial depending on allocation pattern. It can also be switched to CUDA’s own allocator: the backend option accepts cudaMallocAsync, which requires CUDA 11.4 or newer.
CuPy uses its own memory pool by default and offers an experimental interface to the stream-ordered allocator. The key difference, in CuPy’s words, is that CUDA’s pool lives in the driver, so other CUDA code in the same process can allocate from the same pool. CuPy’s MemoryAsyncPool is built on the stream-ordered allocator, and its set_limit is implemented through cudaMemPoolAttrReleaseThreshold.
This matters when a pricing service mixes libraries. NVIDIA’s RMM write-up notes that when several libraries each keep their own pool, they compete for device memory instead of sharing it. CuPy documents a related trap: memory freed to the stream-ordered pool may not be immediately visible to code using cudaMalloc, causing out-of-memory errors until a synchronisation or free_all_blocks(). An external allocator interface, which lets the application plug one allocator into each library, enables that sharing.
CUDA or SYCL for the pricing engine, without betting the rewrite
The low-regret way to choose between CUDA (with cuBLAS and Thrust) and a SYCL stack is to run a short pilot on your own pricing and backtesting kernels, then put the compute backend behind an abstraction layer so the choice can be revisited. TechnoLynx’s GPU Performance Engineering service runs that pilot: it captures the workload’s compute patterns and latency targets, builds representative kernels on both stacks for your cluster, and measures throughput, scaling and the cost of swapping adapters.
CUDA is NVIDIA’s proprietary platform with a mature library ecosystem; SYCL is a vendor-neutral Khronos standard whose library coverage varies more by implementation. The pilot should measure three things on real kernels: throughput, how fast your team develops on each stack, and whether the libraries you need exist. Choose CUDA if you need mature libraries now and your cluster is NVIDIA; choose SYCL if portability to non-NVIDIA accelerators is a stated priority and the pilot’s performance gap is acceptable. Kernels that do not resemble production will bias the result, so they deserve as much care as the measurement.
Whichever wins, a ports-and-adapters design keeps strategy and backtesting logic off backend-specific APIs, so a later switch means writing a new adapter rather than rewriting the backtester. The allocator belongs behind that port too; RMM’s common memory resource interface is a working example of allocation strategy as a pluggable component. If the argument for leaving CUDA is mainly lock-in, the trade-offs of porting a GPU path off CUDA with HIP are worth reading alongside a SYCL pilot. When the pilot raises a hardware question too, LynxBenchAI’s published results show how each device runs the public PyTorch stack; they are broadly indicative, but they do not measure your pricing kernels. Revisit the decision when the SYCL tooling landscape or your hardware roadmap changes materially.
Validating a CUDA driver upgrade before it reaches production pricing
A safe CUDA or driver upgrade validates the candidate version in a shadow environment against the production pricing kernels, compares outputs and latency to the certified baseline, and keeps a tested path back to that baseline.
- Pin the current certified combination of driver, CUDA toolkit, cuBLAS and cuSOLVER as an immutable image.
- Build the candidate image with only the upgraded components changed.
- Run the full pricing regression suite in a shadow environment on the same GPU model, comparing prices and sensitivities against predefined numerical tolerances and checking for latency regressions.
- Promote only if both pass; otherwise revert to the pinned image automatically and investigate.
Allocator behaviour belongs in that suite, because opportunistic reuse can vary allocation patterns from run to run. Feature floors matter too: cudaMallocAsync arrived in CUDA 11.2, and PyTorch’s cudaMallocAsync backend requires 11.4. Telling expected floating-point reduction differences apart from genuine kernel bugs is its own diagnostic job, and it deserves a separate investigation rather than a widened tolerance.
