A vendor pitches “reinforcement learning as a service” the way an object-storage vendor pitches a bucket: send us your environment, we return a trained policy, and you never think about the machine underneath. That framing is convenient, and for the first pilot it is fine. It also hides the one decision that determines your long-run cost — the compute API the service is built on. The decision to buy RL as a service is, underneath, still a GPU compute API decision. That claim sounds like an over-read of a managed product. It is not. Reinforcement learning is training- and simulation-heavy in a way that supervised inference is not, and every part of the loop that matters — rollout generation, environment stepping, policy update — resolves to GPU kernels. Whichever API those kernels are written against sets both your throughput ceiling and your portability. Ignore it and you inherit a vendor-lock cost you never quantified. How does RL as a service work in practice? Strip the marketing and a managed RL platform is three coupled compute stages wrapped in an API. You supply an environment (a simulator, a game, a market model, a robotics rig abstraction) and a reward specification. The service runs an outer loop that alternates between two phases. First, rollout — the current policy acts in many copies of the environment to collect trajectories. Second, policy update — those trajectories feed a gradient step (PPO, SAC, IMPALA-style variants) that produces a new policy. Repeat until the reward curve converges. The abstraction the service sells is that you never see this loop. You see a “submit environment, poll for a trained policy” interface. The abstraction the service cannot remove is that both phases are GPU work, and the ratio between them varies wildly by problem. A robotics policy with a physics-heavy simulator can spend most of its wall-clock in environment stepping; a language-model RLHF run can spend most of it in the policy-update forward/backward pass. Frameworks like SLIME’s approach to asynchronous RL rollouts exist precisely because that phase imbalance, left naive, leaves GPUs idle. So “how it works in practice” is not really about the API surface. It is about where your specific workload spends its GPU-hours — and that determines what a service is actually charging you for. What GPU workload does reinforcement learning generate, and where is the throughput ceiling? Break the loop into its kernel-level components, because that is where the ceiling lives. Loop phase GPU work Where it bottlenecks Rollout / acting Batched policy forward passes; inference kernels (matmul, attention if the policy is a transformer) Small-batch inference latency; poor kernel occupancy when environments are few Environment stepping Simulation kernels (physics, game logic) or CPU-GPU round-trips if the sim is off-device Host↔device transfer stalls; PCIe topology; sim not GPU-resident Policy update Full forward + backward + optimizer step; gradient all-reduce across GPUs HBM bandwidth; NCCL collective cost at scale Data movement Trajectory buffers shuttled between rollout and update workers Memory-copy overhead, buffer contention The throughput ceiling is set by whichever of these dominates your problem, not by the peak FLOPS on the datasheet. This is an observed pattern across the GPU-bound RL work we have seen rather than a benchmarked constant: a training run can show 90% “GPU utilization” on a monitoring dashboard and still be throughput-bound on host-to-device transfers, because the utilization counter registers a busy SM without telling you it is busy waiting on a copy. The practical consequence: two managed RL services quoting the same GPU type can differ by a large factor in GPU-hours per converged policy, purely because one keeps the environment GPU-resident and fuses rollout kernels while the other round-trips through the host. The datasheet is identical. The bill is not. That gap is a kernel-and-API question, and it is invisible from the service’s API surface. Which compute API sits under managed RL platforms, and why does that matter? Here is the part the black-box framing erases. The rollout, stepping, and update kernels are written against some GPU compute API, and today the overwhelming default is CUDA. That is not an accident — the mature RL tooling (the PyTorch training stack, custom simulation kernels, NCCL for multi-GPU gradient reduction, FlashAttention for transformer policies) is CUDA-first, and building a competitive managed RL stack on anything else means porting a large amount of hand-tuned kernel code. The alternatives are real but carry cost. OpenCL and SYCL (the latter via oneAPI) offer cross-vendor compute, and the trade space between them is the same one we walk through in CUDA vs OpenCL when porting AI workloads across GPUs. The uncomfortable truth is that CUDA-specific memory patterns — coalesced access tuned for NVIDIA’s memory hierarchy, kernel fusion assuming a particular occupancy model — do not port performantly even through translation layers. You can get functional portability. You often cannot get performant portability without a genuine re-tune. Why does the API under a managed service matter to you, when you supposedly never touch it? Because it silently decides which hardware your training runs on, and therefore which vendor’s rate card sets your per-episode cost. If the service is CUDA-locked, it runs on NVIDIA capacity, and your cost floor is NVIDIA’s pricing — during a supply crunch, that floor rises with the market. If the service’s kernels port to OpenCL or SYCL, it can schedule the same rollout volume onto AMD (via HIP/ROCm) or Intel accelerators, and it can arbitrage price across vendors. You do not see the API. You see the invoice that the API produced. Does the abstraction hide a vendor lock-in cost, and how do I quantify it? Yes, and the quantification is more tractable than teams assume because RL gives you a natural unit: GPU-hours per converged policy. Work the example with explicit assumptions. Suppose a policy converges in roughly 500 GPU-hours on the service’s default NVIDIA capacity (illustrative — measure your own; convergence GPU-hours vary by reward sparsity and environment cost). If NVIDIA-class capacity prices at, for illustration, $3/GPU-hour and equivalent AMD/Intel capacity is available at $2/GPU-hour for the same rollout throughput, then a portable stack lets you buy the same converged policy for two-thirds the price. Across dozens of policy iterations in an active research program, that delta compounds into the dominant line item. Quantifying RL-as-a-service lock-in: a rubric Establish the unit cost. Measure GPU-hours per converged policy on the service’s default capacity for one representative task. This is a benchmark-class measurement only if you name the task and pin the config; otherwise treat it as an internal planning heuristic. Find the price of the alternative. Get a spot/reserved rate for AMD (ROCm) or Intel (oneAPI) capacity that can sustain the same rollout throughput. Same throughput, not same peak FLOPS. Compute the locked premium. (locked_rate − portable_rate) × GPU-hours_per_policy × expected_policy_count. That number is what the CUDA lock is costing you over the program’s life. Test whether the service can even move. Ask the vendor directly: can this workload run on non-NVIDIA accelerators, and at what throughput penalty? A vague answer is the answer — the stack is CUDA-locked. Discount for switching friction. If you ever wanted to leave the service entirely, the environment/reward code is portable but any custom kernels the vendor tuned are not. Price the re-tune. The number that falls out of step 3 is the figure most procurement conversations never produce, because the service was evaluated on features, not substrate. Can I move an RL workload between NVIDIA, AMD, and Intel without a performance collapse? Sometimes, with caveats that mirror the general porting story. If your RL stack lives at the framework level — PyTorch with standard operators, no hand-written CUDA kernels — then ROCm’s PyTorch build or Intel’s oneAPI path can run it with a re-tune rather than a rewrite, and the porting mechanics are the same ones covered in our broader GPU performance and portability work on the GPU acceleration page. The moment the stack depends on custom fused kernels — a bespoke simulation kernel, a fused rollout-collection path, an attention variant tuned for a specific SM generation — the port stops being free. The honest framing is a spectrum, not a yes/no: Framework-only RL stack → portable with modest re-tune; performance collapse unlikely if you re-profile. Framework + a few custom kernels → portable in principle; expect a real re-tune effort and a throughput penalty until it lands. Deeply CUDA-fused stack (or a managed service built on one) → not portable without engineering that the managed abstraction hid from you in the first place. This is why the “which API sits underneath” question is not academic. A service that answers “framework-level, no custom CUDA” is a fundamentally more portable purchase than one whose speed comes from proprietary kernels — even if both quote the same GPU and the same price today. How do I decide between buying RL as a service and running my own GPU stack? Frame it as a substrate decision, not a build-versus-buy checklist. Managed RL is the right call when your team’s scarce resource is RL engineering time and your policy count is low enough that the locked premium stays small. Running your own stack — on the same PyTorch/ROCm/oneAPI foundations — becomes the right call when policy iterations are frequent, the locked premium compounds, and portability across accelerator vendors is itself a cost lever you want to hold. The mistake is treating the two as different categories. They are the same GPU compute problem with a different owner of the kernels. The generic version of this trade-off — where the infrastructure cost hides behind a clean API — is exactly what we lay out in how machine-learning-as-a-service hides GPU cost, and RL is the sharpest instance of it because the compute intensity is highest. If you are deploying policies to constrained targets rather than training them, the parallel question of runtime selection is covered in choosing RL frameworks for edge inference. FAQ What’s worth understanding about RL as a service first? A managed RL platform wraps three coupled GPU stages — rollout (the policy acts in the environment), environment stepping, and policy update (the gradient step) — behind a “submit environment, receive trained policy” API. In practice what matters is not the API surface but where your specific workload spends its GPU-hours, because the phase ratio varies enormously between, say, a physics-heavy robotics task and a language-model RLHF run. What GPU workload does reinforcement learning actually generate, and where is the throughput ceiling? RL generates batched inference kernels (rollout), simulation or host-transfer work (environment stepping), and full forward/backward/optimizer plus gradient all-reduce (policy update). The throughput ceiling is set by whichever phase dominates your problem — often host-to-device transfers or HBM bandwidth — not by the peak FLOPS on the datasheet. A dashboard can show high GPU utilization while the SM is busy waiting on a memory copy. Which compute API tends to sit under managed RL platforms, and why does that matter for me? The default is CUDA, because the mature RL tooling (PyTorch, NCCL, FlashAttention, custom simulation kernels) is CUDA-first. It matters because the API silently decides which hardware your training runs on: a CUDA-locked service runs on NVIDIA capacity and fixes your per-episode cost to NVIDIA’s rate card, while a stack that ports to OpenCL or SYCL can schedule the same rollout volume onto AMD or Intel and arbitrage price. Does the RL-as-a-service abstraction hide a vendor lock-in cost, and how do I quantify it? Yes. Use GPU-hours per converged policy as the unit: measure it on the service’s default capacity, price the same rollout throughput on AMD or Intel capacity, and compute (locked_rate − portable_rate) × GPU-hours_per_policy × expected_policy_count. That figure is the lock-in premium most procurement evaluations never produce because they scored features rather than substrate. Can I move an RL training workload between NVIDIA, AMD, and Intel accelerators without a performance collapse? It depends on how deep the CUDA dependency runs. A framework-only PyTorch stack ports to ROCm or oneAPI with a re-tune rather than a rewrite; a stack with a few custom kernels ports with real effort and a throughput penalty; a deeply CUDA-fused stack does not port performantly, because CUDA-specific memory patterns do not carry over even through translation layers. How do I decide between buying RL as a service and running my own GPU RL stack for cost and portability? Treat it as one GPU compute problem with a different owner of the kernels, not two categories. Managed RL fits when RL engineering time is your scarce resource and policy count is low; running your own stack fits when policy iterations are frequent, the locked premium compounds, and cross-vendor portability is a cost lever you want to hold yourself. What should a GPU performance audit check before I commit an RL workload to a managed service? An audit treats the service’s compute API as a first-class dimension: it checks whether the GPU substrate is structurally suboptimal or lock-in-prone for your rollout and training profile, whether the phase imbalance leaves GPUs idle, and whether the stack could move to non-NVIDIA capacity at an acceptable throughput penalty. The question worth carrying out of any managed-RL evaluation is not “does the service train a good policy” — most will. It is “if NVIDIA capacity doubles in price next quarter, what does my per-policy cost do, and can this substrate move?” A GPU performance audit exists to answer exactly that, by treating the compute API under the service as a dimension you can measure rather than a black box you agreed to. The teams that ask it before signing keep a portability option the black-box framing quietly sold away.
