A reinforcement learning framework is usually chosen on training ergonomics — which one has the cleanest API, the most algorithms, the best logging. That choice quietly decides something else entirely: what your trained policy looks like when it has to ship. The framework is not just a training convenience. It is an upstream deployment decision that determines your policy’s architecture size, its export path, and whether it survives compression to CoreML, ONNX Runtime, or WebGL without a per-target re-validation cycle. The naive assumption is that the framework’s job ends when training converges. It does not. Two frameworks can produce policies that are equally accurate on your evaluation environment and yet diverge completely at the deployment boundary — one exports to a portable, distillable graph you can ship everywhere, and the other locks you into runtime-specific artifacts you have to babysit per platform. If your policy is going to run on phones, in a browser, and on an embedded runtime, that divergence is the decision that actually matters. How do RL frameworks work, and what does that mean in practice? An RL framework wraps three things: an environment interface, an algorithm implementation (PPO, SAC, DQN, and their variants), and a training loop that repeatedly samples trajectories, computes an update, and steps a policy network. Stable-Baselines3 sits on top of PyTorch and gives you a small, opinionated set of well-tested algorithms. RLlib, part of Ray, scales rollouts across a cluster and supports both PyTorch and TensorFlow backends. CleanRL takes the opposite stance — single-file implementations you read top to bottom. For training, those differences are about throughput, reproducibility, and how much of the loop you can see. For deployment, only one thing matters: the policy network the framework hands back. That network is a plain neural net — usually a small multilayer perceptron for low-dimensional control, or a convolutional or transformer encoder for pixel and sequence inputs. What differs across frameworks is not the math but the packaging: how the policy is stored, what custom ops the framework injected around it, and how cleanly you can lift the forward pass out of the training harness. In our experience, the teams that get burned at deployment are the ones who never inspected the exported graph until the sprint they needed to ship it. By then the architecture is frozen and the framework’s export quirks are yours to work around. How the framework shapes the policy you eventually deploy The framework influences architecture in ways that are easy to miss during training. Default policy networks are sized for training stability, not inference budgets — a framework that defaults to a wide two-layer MLP produces a policy that is trivially small, while one that wraps a recurrent or attention-based encoder produces something that quantises and fuses very differently on an edge runtime. Custom action-distribution layers, observation normalisers baked into the graph, and framework-specific preprocessing all travel with the exported model unless you strip them out deliberately. This is the same cross-target logic that governs any compression strategy: what you train constrains how the architecture can span your target matrix. A policy built from standard nn.Module layers exports to ONNX cleanly and distils into a smaller student without drama. A policy that depends on a framework’s custom sampling op, or on a Python-side normaliser that never made it into the graph, will either fail to export or export into something that behaves differently once the surrounding harness is gone. The practical rule: treat the policy architecture as a deployment artifact from day one. If you know the target is CoreML on iOS, ONNX Runtime on a Linux edge box, and WebGL in a browser, you want a forward pass that reduces to standard convolutions, linear layers, and activations — the operator set every runtime supports. This is where a clear-eyed view of what to measure on multi-platform edge pays off before you commit to a training run. Which RL frameworks export cleanly to ONNX, and why it matters ONNX is the interchange format that decouples your training framework from your deployment runtime. If your policy exports to clean ONNX — a graph built only from standard operators — you can load it in ONNX Runtime, convert it to CoreML, or compile it toward a WebGL backend from a single source of truth. If it does not, you maintain a separate export path, and often a separate model, per platform. Frameworks built directly on PyTorch tend to export well, because torch.onnx.export traces the policy’s forward pass into a standard operator graph. The failure modes are consistent and worth naming: control flow that traces into fixed branches, custom autograd functions with no ONNX symbolic, and framework preprocessing that lives in Python rather than in the module. TensorFlow-backed policies go through tf2onnx, which works but adds a conversion hop that can surface its own operator gaps. RL framework export-fit at a glance The table below is a directional planning aid, not a benchmark. Export behaviour depends on your exact policy architecture and framework version; treat it as a starting hypothesis to validate against your own graph. Framework Backend ONNX export path Edge-fit signal Stable-Baselines3 PyTorch torch.onnx.export on the extracted policy Good — small MLP/CNN policies export to standard ops CleanRL PyTorch Direct export of a plain nn.Module Good — single-file policies are easy to isolate and trace RLlib PyTorch / TF torch.onnx or tf2onnx per backend Mixed — scale-focused; export needs deliberate policy extraction Tianshou PyTorch torch.onnx.export on the network module Good — modular networks separate cleanly from the trainer The signal to read here is not “which framework is best” but “which framework produces a forward pass I can lift into a standard graph.” That single property decides whether one policy spans your target matrix or whether you inherit N export pipelines. For teams weighing a managed path instead of running training in-house, our discussion of how managed reinforcement learning works and its GPU API trade-offs covers where the abstraction helps and where it hides the export problem. Should you distil or quantise a trained RL policy before shipping? Both, usually, and in that order — but the choice is conditional. Distillation trains a smaller student policy to reproduce the teacher’s action outputs, giving you architecture control the original training run may not have delivered. Quantisation reduces numeric precision (typically FP32 to INT8) to shrink the model and speed up inference on hardware that supports it. They solve different problems, and the deployment target decides which you lean on. A distilled policy built from standard operators ships to all targets with consistent quality; you validate the student once and deploy it everywhere. A per-platform quantised graph is the opposite — quantisation that ONNX Runtime accepts may need re-tuning for CoreML, and WebGL backends often prefer FP16 or FP32 entirely. That is the divergence the framework choice set in motion at training time. A decision rubric for the compression path Policy already small and standard-op? Skip distillation; go straight to per-target quantisation checks. The forward pass is your portable artifact. Policy oversized or full of framework-specific ops? Distil first into a clean student, then quantise. Distillation is where you fix the architecture the framework handed you. Target set includes WebGL? Assume FP16/FP32 in the browser and validate INT8 only where the runtime supports it. Do not assume one quantised graph covers all three. Latency budget tight on the smallest device? Distil to that device’s constraint first, then relax for larger targets — sizing up is cheaper than sizing down. We treat this as an observed pattern across edge deployment work, not a benchmarked rule: the projects that ship one distilled, standard-op policy avoid roughly three times the per-platform validation churn that runtime-specific quantised graphs incur across CoreML, ONNX Runtime, and WebGL. The number is a planning heuristic — your target matrix and quality tolerance move it — but the direction is reliable. One portable policy beats N maintained-and-re-tested compressed models. What tradeoffs appear when one policy runs across several edge runtimes? Quality and latency both shift, and they shift for different reasons. Quality drift comes from precision: an INT8 policy that is indistinguishable from FP32 on ONNX Runtime can degrade measurably in a WebGL backend that emulates or rounds differently. Latency shifts come from operator coverage — a runtime that lacks a fused kernel for one of your layers falls back to a slower path, and your carefully profiled inference time on one target does not transfer to another. The failure this produces is subtle: the policy passes evaluation on your development runtime, then behaves worse in the field on a different one, and nobody isolates the runtime as the cause because “the model is the same.” It is not the same. It is the same graph running through different operator implementations at different precisions. Naming the runtime as a variable in your evaluation is the fix, and it connects directly to how multi-platform edge compilation flags can quietly change numeric behaviour between builds. How to evaluate an RL framework against your deployment target matrix Before you commit to a training run, do a small export dry-run. Build a minimal policy in the candidate framework, export it to ONNX, and try loading it in ONNX Runtime, converting it to CoreML, and compiling it toward your browser target — before you have anything worth deploying. This costs an afternoon and surfaces every custom-op and preprocessing gap while they are still cheap to fix. The decision to make is not “which framework has the best training UX” but “which framework produces a policy that spans my target matrix from a single source of truth.” Rank your candidates on export cleanliness, architecture control, and whether the forward pass reduces to standard operators — then let training ergonomics break the tie. You can validate whether the model your chosen framework produces is compatible with your multi-platform compression strategy with an [A1 GPU/inference optimization assessment](GPU engineering), which evaluates the export graph and architecture complexity against your edge target set. The broader engineering context for that work lives on our GPU engineering practice page. FAQ How do RL frameworks work, and what does it mean in practice? An RL framework wraps an environment interface, an algorithm implementation such as PPO or SAC, and a training loop that samples trajectories and updates a policy network. Frameworks differ in throughput, reproducibility, and how much of the loop you can inspect. For deployment, the only thing that matters is the policy network the framework hands back and how cleanly you can lift its forward pass out of the training harness. How does the choice of RL framework affect the architecture and size of the policy model you eventually deploy? Default policy networks are sized for training stability, not inference budgets, and custom action-distribution layers, observation normalisers, and framework preprocessing travel with the exported model unless you strip them out. A policy built from standard nn.Module layers exports cleanly and distils into a smaller student; one that depends on a framework’s custom ops may fail to export or behave differently once the harness is gone. Treat the policy architecture as a deployment artifact from day one. Which RL frameworks export cleanly to ONNX, and why does that matter for multi-platform edge targets? Frameworks built directly on PyTorch (Stable-Baselines3, CleanRL, Tianshou) tend to export well through torch.onnx.export, because the forward pass traces into standard operators. TensorFlow-backed policies go through tf2onnx, which adds a conversion hop that can surface operator gaps. Clean ONNX export matters because it lets one policy load in ONNX Runtime, convert to CoreML, and compile toward WebGL from a single source of truth instead of N per-platform pipelines. Should I distil or quantise a trained RL policy before shipping it to CoreML, ONNX Runtime, and WebGL? Usually both, in that order, but conditionally. Distillation trains a smaller student to reproduce the teacher’s actions and gives you architecture control; quantisation reduces precision to shrink and speed the model on supporting hardware. If the policy is already small and built from standard operators, skip distillation and go straight to per-target quantisation checks; if it is oversized or full of framework-specific ops, distil first into a clean student, then quantise. What quality and latency tradeoffs appear when a policy trained in one framework runs across several edge runtimes? Quality drift comes from precision — an INT8 policy that matches FP32 on ONNX Runtime can degrade in a WebGL backend that rounds differently. Latency shifts come from operator coverage, where a runtime lacking a fused kernel falls back to a slower path, so profiled timings on one target do not transfer. The subtle failure is a policy passing evaluation on the development runtime and behaving worse in the field on a different one, so name the runtime as a variable in evaluation. How do I evaluate an RL framework against my deployment target matrix before committing to training? Do a small export dry-run: build a minimal policy in the candidate framework, export to ONNX, and try loading it in ONNX Runtime, converting to CoreML, and compiling toward your browser target before you have anything worth deploying. Rank candidates on export cleanliness, architecture control, and whether the forward pass reduces to standard operators, then let training ergonomics break the tie. The framework you pick on training day is the framework that decides, months later, whether one policy spans every edge target or whether you inherit a validation pipeline per platform. That is the question to answer before the first training run, not after convergence — because what you train constrains how the architecture can span your target matrix.
