We would give pricing, risk and training a guaranteed floor each on a shared quant GPU cluster by combining two layers. The first is an entitlement layer: Slurm QOS caps plus recurring reservations, or a Kueue ClusterQueue’s nominal quota. The second is a slack layer of backfill, bounded preemption and a held-back idle pool, which lets batch work use whatever the floors leave idle. The same toolkit serves every workload class, but the policy built from it differs for each. A deadline-bound VaR run, an intraday re-pricing service, a fat-tree procurement and an audit request each need their own allocation unit and their own evidence.
For trading and risk teams this work sits inside wider AI engineering for finance.
Guaranteed minimum GPU shares for pricing, risk and training
We would give each workload class a written entitlement that the scheduler enforces as a reservation or quota. Everything above that floor is shared through backfill and preemption rules. In Slurm, the QOS attached to a job sets three things: its scheduling priority, its preemption rights and its resource limits.
TechnoLynx designs this policy layer starting from the workload mix. It frames each class’s deadline and isolation needs and picks Slurm, Kubernetes or a hybrid. It then implements the guaranteed minimums on the target cluster and load-tests overlapping job classes to show that none of them starves.
Slurm, Kubernetes, or both for mixed pricing, risk and training?
The workload’s shape decides it. Pure batch work with deadlines and multi-node jobs fits Slurm, which has powered TOP500 systems and commercial backends such as AWS. Meta’s two ML research clusters described at HPCA 2025 (16k and 8k GPUs) run it on bare metal, with GPU quotas set per project. For containerised, multi-tenant work, KAI Scheduler on Kubernetes provides GPU sharing and consolidating placement. Where both kinds of work exist, Slurm for batch and Kubernetes for services is a defensible split. The cost is two schedulers and two monitoring stacks to run.
Committed shares in Slurm: QOS, GrpTRES, reservations and backfill
QOS affects priority and preemption only when two settings are in place. It affects priority only if the multifactor plugin is loaded and PriorityWeightQOS is non-zero, and it decides preemption only when PreemptType=preempt/qos. With preempt/qos set, a queued job whose QOS may preempt another QOS will displace running jobs of that QOS. Limits also need AccountingStorageEnforce=limits in slurm.conf. Without it, limits can be configured but are not enforced.
The cap is GrpTRES. Once a QOS reaches its total TRES count, new jobs queue until resources are released. A QOS limit applies across every partition that uses the QOS, not separately to each one. A cap is a ceiling, though, and a floor needs one more mechanism. One option is a partition QOS with GrpTRES on a partition spanning all nodes. That gives a “floating” partition that receives a fixed amount of resources without being tied to particular nodes.
The other option is a recurring reservation. Reservations can be granted to users, accounts, QOS or partitions. They can repeat DAILY, and they are re-validated each time they advance to the next window. A job with a reservation is considered before any other job in the same partition. The MAGNETIC flag pulls in jobs that meet the reservation’s access controls even when they do not name it. Reserved cores and nodes are exclusive to the reservation’s jobs unless the FLEX flag is set.
Backfill is the mechanism we would use for the slack. Slurm starts a lower-priority job only if that does not delay the expected start of any higher-priority job. This depends on jobs carrying real time limits: without them every job has the same limit and backfilling is impossible.
The starvation risk is specific. The backfill window should be at least as long as the highest allowed time limit to prevent starvation. A NoReserve QOS that cannot be preempted by every other QOS can starve larger jobs.
Priority tuning is worth measuring rather than assuming. In a 2025 arXiv study, RLTune, a reinforcement-learning and MILP scheduler layered on Slurm, cut makespan by 24.8% and average wait time by 30.5% against the multifactor baseline on a 1,024-job synthetic trace.
The Kubernetes version: Kueue nominal quota, KAI queues and GPU fractions
In Kueue, the guaranteed minimum is a ClusterQueue’s nominalQuota. borrowingLimit and lendingLimit control how much unused quota moves between ClusterQueues in the same cohort. Set to LowerPriority or Any, reclaimWithinCohort lets a pending workload that fits its nominal quota preempt cohort workloads running above theirs, and LowerPriority limits this to lower-priority ones. The default value, Never, does not preempt them. Fair Sharing preemption goes further: a ClusterQueue can preempt until it holds an equal or weighted share of the borrowable resources.
KAI Scheduler expresses the same idea through queues. Each queue has a guaranteed quota, an over-quota weight for surplus and a hard limit. At the level of a single GPU, CUDA 13.4 fractioning lets a workload declare a guaranteed GPU-memory request plus a burst maximum. KAI Scheduler’s NvFractions mode writes the annotations that the GPU operator enforces on the node. For how to subdivide GPUs between tenants, see matching GPU bin-packing and multi-tenant scheduling to workload shape.
Two Kubernetes caveats change the design. First, ResourceQuota does not guarantee capacity: a team within its quota can still fail to schedule on a full cluster. Second, too many high-priority PriorityClasses take away the scheduler’s ability to tell workloads apart.
Can gang scheduling guarantee risk jobs contiguous NVLink domains while training runs?
With the topology/block plugin, yes, at the cost of exclusive nodes and some fragmentation. The topology/tree plugin cannot guarantee that a job’s nodes land under a single leaf. The block plugin does guarantee placement according to the defined topology. Block topology puts minimising fragmentation ahead of first-fit placement. It only places a job on adjacent blocks, and it assumes nodes are allocated exclusively. The –segment option mitigates that fragmentation and lets users trade absolute performance for quicker scheduling.
In trace-driven simulation, Dally’s delay scheduling, which relaxes strict consolidation, cut makespan by 69% against a consolidation-based policy.
The terms need care in a design document. In Slurm, “gang scheduling” means time-slicing several jobs on the same resources, and all of those jobs must fit in memory at once. Distributed ML jobs need something different: all of their resources allocated at the same moment.
Node counts, GPU-hours or priority weights: which guarantee is easiest to audit?
Node or GPU counts are the easiest to audit, GPU-hours come next, and priority weights are the hardest. Weights are hard because Slurm’s fair-share factor is not a fixed allotment. It orders queued jobs by how under- or over-served their accounts are, so no figure in the configuration states what a class actually received.
| Guarantee unit | How it is expressed | What an auditor can check | Fits |
|---|---|---|---|
| Node or GPU count | Reservation NodeCnt or TRESPerNode, GrpTRES cap, Kueue nominalQuota | One number, comparable with accounting at any moment | Deadline classes: VaR, EOD pricing |
| GPU-hours | TRES-minutes limits per association or QOS | Consumption over a period against entitlement | Research and training budgets |
| Priority weight | PriorityWeightQOS, fair-share, over-quota weight | Only relative ordering, never a delivered amount | Sharing surplus above the floors |
To set the size of each class’s entitlement, we would start from measured peak demand inside the deadline window, not from headcount or budget share. One published rule for Kubernetes namespace quotas (SquareOps) is to measure 30 days of usage and set the request quota at about 1.5 times the observed peak. The same rule lets the sum of quotas exceed cluster capacity, because teams do not all peak at once. We would not apply that over-subscription to deadline classes. We would size VaR and pricing floors to fit inside physical capacity together, and over-subscribe only training floors.
Handling a one-off backtest without reopening the allocation policy
A separate low-priority, preemptible QOS takes in one-off work such as a new strategy’s backtest, so the base policy never has to be renegotiated. MIT SuperCloud runs this pattern as “spot” jobs. These can exceed users’ normal limits and can be preempted by any interactive job. We would also give the burst QOS a minimum run time before it can be preempted, so its jobs make progress. Slurm’s PreemptExemptTime sets this. Meta’s clusters apply a similar rule: jobs become eligible for preemption only after two hours.
What happens if both teams need the full cluster the same night?
The floors decide, and the policy has to state in advance which one wins in a real collision. Slurm evaluates jobs in a fixed order: jobs that can preempt, then jobs with a reservation, then partition PriorityTier, then job priority. That order, with the floors’ sizes and preemption rules, is set in configuration before the night. If the floors together add up to more than the cluster, no configuration fixes it. At that point the problem is capacity or contracts, not scheduling.
Signing off fat-tree capacity and latency before procurement
We would put a fat-tree to sign-off only when the model states its inputs explicitly: traffic profiles, hop counts and oversubscription ratio. It should pair an analytical bound with discrete-event simulation for bursty contention, and add a physical testbed where the stakes justify the cost. How much rigour the committee accepts is a judgement about the cost of being wrong, not a fixed rule. We would base capacity sign-off on profiled workload needs rather than spec-sheet figures.
Classic fat-tree latency models derive the average hop count from the tree structure under uniform traffic and add contention terms. One course analysis builds such a model, drawing on Dally’s k-ary n-cube example.
The oversubscription ratio is where cost and bandwidth trade off. Al-Fares et al. (SIGCOMM 2008) note that oversubscription is usually introduced to lower total cost, and that a 1:1 ratio can be hard to achieve in practice. Full bisection bandwidth is not automatically the best design either. A bottleneck-structure analysis (QTBS) showed, in an analytic example under congestion control, that a three-level binary fat-tree tapered at 4/3 minimised the completion time of the slowest flow, while the full fat-tree was inefficient under congestion control.
Failure belongs in the model too. Meta’s HPCA 2025 cluster study reports that without resilience mechanisms, more than 50% of bandwidth may be lost. A model that only covers a healthy fabric therefore says nothing about the night a link flaps.
| Method | What it gives the committee | Blind spot | Use when |
|---|---|---|---|
| Analytical queueing model | Fast bounds from hops, oversubscription and load | Assumes uniform traffic, while real GPU cluster load is bursty | Traffic is well characterised and a bound is enough |
| Discrete-event simulation | Contention, burst and tail behaviour | Only as good as the traffic profile fed in | Tail latency or burst overlap drives the SLA |
| Physical testbed | Empirical confirmation on real switches and NICs | Costly, may not reach full fan-out | Stakes are high or the committee needs measured proof |
Burstiness is why we add simulation. Workload pressure in GPU clusters is non-stationary: some windows stay nearly idle while others pile up hundreds of hours of cumulative wait. For the model to survive later scrutiny if an SLA is missed, the pack should record:
- the traffic profiles used and where they came from (profiled jobs, trace replay or assumption)
- hop counts per path class, and the oversubscription ratio at each tier
- the fan-out pattern tested, and whether it reached full scale
- degraded-fabric cases (a lost link, a lost spine)
- which assumptions a testbed confirmed and which remain modelled only
Who should be allowed to change Slurm queue priorities and reservations?
Cluster administrators or a change board should change QOS priorities, limits and reservations, and quant teams should only submit jobs. Any change that could affect another team’s risk run needs sign-off from someone other than the requester, recorded in a retained approval log. Slurm already draws part of that line: creating, updating and deleting reservations are privileged scontrol operations.
TechnoLynx maps who can change scheduler settings today and designs the role model and approval workflow. It connects logging to Slurm accounting or ticketing, and tests that a pricing-side change request cannot take effect without recorded risk-team sign-off.
The control matters because Slurm applies scheduling policy changes immediately. Policy changes are uploaded to the control daemons and take effect at once, and deleting an association cancels its running and pending jobs straight away. Some loopholes need closing explicitly. An unprivileged user can delete a reservation they have access to if user_resv_delete is set in slurm.conf, or if the reservation carries the USER_DELETE flag.
That answers the question of how to stop a busy pricing run from quietly starving a risk run. The GrpTRES cap and the risk floor stop a legitimate pricing burst from eating risk capacity, and change control closes the other route: an unreviewed edit to the policy itself.
The audit artefact maps onto an established control. 5 control CM-3 requires an organisation to document which types of change are configuration-controlled, to review and approve or reject them with explicit impact analysis, and to keep records for a defined period. It names Configuration Control Boards or Change Advisory Boards as the review mechanism. Etz et al. (2025) describe HPC change management that adds risk assessment, extra testing and fallback provisions for changes that could affect time-sensitive services. They also describe formalised justification processes for requesting prioritised access.
The pack handed to the audit office has three parts. The first is the policy itself, listing which QOS, partition and reservation fields are controlled. The second is the approval log, recording requester, approver, what changed, timestamp, justification and expected effect on other classes. The third is Slurm’s per-job accounting records, collected for every job and job step.
This process has limits. It governs scheduler-level actions, not the contention that legitimate approved jobs create. It only works if roles are enforced in Slurm and the operating system, not just on paper. And the audit office may require a retention period or log format of its own.
Will intraday re-pricing crowd out the overnight VaR and stress batch?
The answer comes from a contention test at worst-case overlap, which shows whether shared hardware holds both deadlines or one workload needs its own capacity.
TechnoLynx produces that evidence through its GPU Performance Engineering work. The first step is to define both workloads’ deadlines and latency targets. The next is to profile and break down end-to-end latency under a simulated worst-case peak, which yields the throughput and latency capacity model.
Published GPU-sharing studies show the mechanism, though not for intraday re-pricing against an EOD VaR batch. Strati et al. (Orion, EuroSys 2024) show that temporal sharing gives a latency-sensitive inference job high tail latency, because each request waits for the ongoing training iteration to finish. CUDA streams and MPS raise aggregate throughput without prioritising the latency-sensitive job. Most NVIDIA GPUs also do not let users preempt a kernel once it has been submitted. Orion kept a high-priority inference job’s p99 latency within 14% of ideal while it shared the GPU with training. Orion, though, is a per-GPU scheduler evaluated on DNN workloads, not on pricing kernels.
At node level, MIT SuperCloud’s spot-job design gives a working pattern. It keeps a fixed number of nodes idle for incoming interactive jobs, and a cron script requeues preemptible jobs last-in, first-out while adjusting their MaxTRESPerUser cap. The team found scheduler-driven preemption unsuitable for production because it degraded interactive job scheduling time. With the cron approach, interactive scheduling performance barely changed whether spot jobs were running or not. The price is a pool of nodes kept idle at all times. SuperCloud measured this on interactive job launch, not pricing work. Casting intraday re-pricing as the interactive class and the stress grid as the spot jobs is our design choice, untested on quant workloads.
We would report the worst-case queueing delay in two parts. One is the p99 wait of an intraday request while the batch is at its own peak. The other is the batch’s completion margin against its deadline with intraday running at full rate. We would not rely on an average-load test, since the Philly and Helios GPU traces show bursty load. A one-off study also goes stale as volumes grow, so the model needs a re-validation cadence and production alerting on queueing delay and SLA margin.
Who can build a contention model for our intraday and nightly risk workloads?
TechnoLynx builds and validates the throughput and latency model that tells a desk whether intraday re-pricing and the overnight VaR batch can share GPUs. The work follows the order of the question. It defines both workloads and their deadlines and chooses the scheduling and profiling approach. It optimises the pipeline and kernels to meet deadlines under contention, then tests at simulated worst-case peak and hands over monitoring of queueing delay and SLA margin. This runs as a GPU Performance Engineering engagement. If the answer is separate capacity, LynxBenchAI publishes results showing how each device runs the public PyTorch stack, which is broadly indicative for custom PyTorch models. It does not run or measure the desk’s own pricing workload.
The engagement suits a team whose deadlines are contractual or regulatory and who cannot spare people to build a worst-case harness. It suits less well a cluster small enough that buying separate intraday nodes costs less than the study. An in-house platform team that already runs Slurm or Kubernetes and has GPU profiling experience is a realistic alternative route.
What keeps an overnight batch safe when it runs late or a node fails?
The fallback has to be agreed before the night. It has two parts: a restart path that limits lost work, and a reduced output the desk has already accepted.
On restart, a single task failure in a gang-allocated job can force re-allocation of the whole job. Meta’s clusters automatically requeue jobs killed by failed health checks under the same job ID, and hourly checkpoints are typical for their larger jobs. On scheduling, ResvOverRun sets how long a job may keep running after its reservation ends. The default is zero, meaning it is killed immediately. The FLEX flag lets jobs start early, run past the reservation and use nodes outside it.
We would treat a reduced scenario set, with core VaR scenarios first and the full stress grid later, as a business decision the risk function signs off in advance. The scheduler’s part is to run the priority scenarios in the highest QOS so they finish first. How the batch itself is split to fit its window is covered in scheduling and sharding for deadline-critical risk runs.
Reserved, burst or dedicated GPU capacity
Is reserved plus burst safer than one fixed reservation while real node needs are unknown?
Usually yes, provided the reserved part covers the deadline classes’ measured floor and the burst part can actually be obtained on the night. A fixed reservation has a visible cost. Slurm charges unused reserved resources to the eligible users or accounts as if they had been used. Cloud burst can also be booked ahead rather than hoped for: AWS Capacity Blocks reserve GPU instances up to eight weeks in advance, with up to 64 instances per block. The wider trade-off between committed and pooled capacity is set out in when guaranteed capacity beats shared pools.
When is it cheaper to buy dedicated GPU nodes for a critical job?
Dedicated nodes win in three cases: when a critical job’s floor is already most of what the shared cluster can give it, when its isolation needs rule out sharing anyway, or when a missed deadline costs more than the nodes sitting idle between runs. Jobs that need guaranteed NVLink-block placement already pay part of that price on a shared cluster, because block scheduling assumes exclusive nodes and fragments the cluster. Sharing wins when the classes’ floors rarely bind at the same time.
Rebalancing committed allocations from measured utilisation
We would keep committed allocations fair by pairing a quota-enforcing scheduler with long-horizon usage data, reviewed on a fixed cadence. That data covers GPU-hours consumed against entitlement, queue wait distributions, and idle or fragmented capacity.
TechnoLynx sets up hierarchical queues, preemption and backfill rules and accounting integration on the cluster. It then builds utilisation dashboards and a recurring rebalancing review, and hands over the operational runbooks.
Slurm can collect accounting data for every job and job step, and its accounts can be nested to any depth to mirror departments and desks. PriorityDecayHalfLife controls how long past usage counts towards fair-share, and the default is seven days. Getting that setting wrong is one of the quieter ways a class builds up an unfair backlog.
Allocated time and used time have to be measured separately. Reservation accounting books idle reserved capacity as used, and NVIDIA’s KAI Scheduler material notes that some researchers grab more GPUs than they need early in the day to be sure of availability. For alerting, the SquareOps guide on Kubernetes quotas recommends warning at 80% quota utilisation, so teams get notice before deployments start failing.
The one night that tests the whole design
Every layer above can be checked with a single rehearsal: replay a night with all classes at peak, intraday re-pricing at full rate and one node failure injected partway through. We would call the design defensible if VaR and pricing finish with margin, intraday p99 holds, the burst queue gives way without cascading preemptions, and the change log shows no unapproved edits. If any of those fail, the failure points to which layer needs work: the floor, the slack, the isolation or the governance. A rehearsal still leaves one thing open. It shows the policy holds at today’s volumes, but not how much growth it can absorb.
