A team under latency pressure decides to rewrite its inference path in C++. Three weeks later the p99 has barely moved, because the bottleneck was never Python. This is the most common way a port spends real engineering and buys nothing: the rewrite happened before anyone profiled. The instinct is understandable. Python feels slow, the interpreter has a reputation, and a C++ or WebAssembly target promises a clean, fast runtime. So the team rewrites the inference path in the target language and benchmarks afterwards. The problem is that this sequence answers the wrong question. It tells you how fast the C++ version runs. It does not tell you whether the thing you ported was ever the constraint. A port decision is not a language preference. It is a decision that lives downstream of an honest latency, footprint, or unit-cost target, and it should be made against a profiling baseline — not against a hunch about interpreter overhead. What a Port Actually Has to Beat When an inference request is slow, the wall-clock time decomposes into roughly three buckets: model compute (the matrix multiplies, attention kernels, convolutions that run on GPU or CPU), Python-level overhead (the interpreter, object allocation, the glue code that marshals tensors and dispatches calls), and IO (loading inputs, serialization, network round-trips, disk reads for weights or features). A port to C++ or WASM primarily attacks the second bucket. It can shrink the glue, eliminate interpreter dispatch, and tighten memory layout. What it does not do is make a cuDNN convolution or a FlashAttention kernel run faster — those kernels are already compiled native code, called the same way from Python or C++. If your latency budget is dominated by model compute, rewriting the calling code in C++ moves the engineering cost without moving the bottleneck. That is the failure mode the whole decision exists to avoid. This is why the order of operations matters. Profile first, attribute the time, then decide. A port-or-not decision grounded in a profiling pass closes the gap between the target and reality; a port that bypasses profiling just relocates the same wall-clock time into a more expensive-to-maintain codebase. How Do We Profile the Python Path Before Committing to a Port? The profiling pass has one job: attribute the latency or footprint budget to model compute, Python overhead, or IO, with enough confidence that the port decision is defensible rather than aspirational. A practical sequence: Establish the target, not just the symptom. “It feels slow” is not a target. “p99 inference latency under 40 ms at 200 requests/second” is. The target sets the bar a port has to clear to be worth its cost. Profile against that target under realistic load. A single-request profile in a notebook hides queueing, batching, and contention effects. Use a sampling profiler (py-spy, cProfile with care, or the PyTorch profiler) on the path as it runs under representative concurrency. Attribute time to the three buckets. The PyTorch profiler and torch.profiler traces will separate CUDA kernel time from CPU-side Python time. If the GPU is busy 85% of the wall clock, model compute is your bottleneck and a port will not help. If the GPU is idle most of the request while Python marshals data, you have found a candidate the port can actually move. Quantify the expected gain. Estimate how much of the Python-overhead bucket a port could realistically remove. If Python overhead is 8% of a request and a port removes most of it, the ceiling on your improvement is roughly 8% — almost never worth a full rewrite. In our experience, the single most common surprise in this pass is discovering that the GPU is starved, not the CPU. The inference path spends its time waiting on data loading or a synchronous preprocessing step, and the right intervention is an async data pipeline or a batching change — not a language port. When the gain ceiling is set by IO or kernel time, the decision to restructure the algorithm or the data flow usually beats a rewrite by a wide margin. A Port-or-Don’t-Port Decision Table The decision turns on which bucket dominates and what target you are missing. The table below is the core rubric. Treat the gain estimates as illustrative framing tied to the bottleneck attribution, not as benchmarked figures — the real number comes from your own profiling pass. Dominant bottleneck (from profiling) Target you’re missing Right intervention Port likely to pay off? Model compute (GPU/CPU kernels) Latency / throughput Quantization, kernel fusion, better runtime (TensorRT, ONNX Runtime) No — kernels are already native Python interpreter + glue overhead Latency, esp. high request rate Cython / C-extension, or C++/Rust serving layer Yes, if overhead is a large share IO / data loading / preprocessing Throughput, GPU starvation Async pipeline, batching, prefetch No — port doesn’t touch IO Process footprint / cold start Memory, edge / serverless deploy C++/WASM compiled binary, smaller runtime Yes, footprint is a port-class win Browser / client-side execution Deployability (no server round-trip) WASM via Pyodide or a compiled module Yes — this is what WASM uniquely enables The right-most column is the honest answer most of the time: a port pays off when the overhead it removes is a large share of a budget you are actually missing, or when the target is footprint or client-side execution — outcomes a port delivers that a faster kernel cannot. Everywhere else, the cheaper interventions win. When Does a Partial Path Beat a Full Rewrite? The choice is rarely binary between “stay in Python” and “rewrite everything in C++.” Between those poles sit cheaper options that capture much of the gain at a fraction of the cost and maintenance burden. Cython or a targeted C-extension can close the inference gap without a full port when the overhead is concentrated in a hot loop — a preprocessing step, a tokenizer, a feature transform — that you can compile in place while keeping the rest of the system in Python. This preserves the Python deployment story and the team’s ability to maintain the code, which is the cost a full C++ rewrite quietly imposes for years afterward. The WASM question is different in kind. Choosing a WebAssembly path is usually not about raw speed — native CUDA will beat WASM on compute every time. It is about where the inference runs. Running Python inference in WASM through Pyodide earns its cost when the deployment target is the browser or an edge sandbox and the alternative is a server round-trip you cannot afford or cannot secure. The mechanics of how WebAssembly executes ML workloads explain why WASM trades some compute throughput for portability and isolation — a trade that only makes sense when portability is the target, not speed. There is also the runtime layer to consider before any language port. The inference engine you choose shapes the port decision: if model compute is your bottleneck, moving from a naive PyTorch eager loop to TensorRT, ONNX Runtime, or a torch.compile graph often delivers the gain you were hoping a C++ port would provide — without leaving Python at all. What a Port Costs After You Ship It The benchmark on day one is the seductive number. The cost that gets underweighted is everything after. A ported inference path carries a maintenance tax: a second toolchain, a build pipeline that has to stay green, a smaller pool of engineers who can safely change it, and a divergence risk where the C++ path drifts from the Python reference it was meant to mirror. When the model updates — new weights, a new preprocessing step, a changed output schema — both paths have to move together, and the ported one moves slower. This is real engineering cost that no first-day benchmark captures, and it should sit in the decision alongside the expected gain. The honest accounting compares three numbers: the expected gain in the target language (bounded by the profiled overhead bucket), the one-time port engineering cost, and the avoided cost of porting an inference path where Python was never the bottleneck. That third number — the cost you don’t pay by deciding not to port — is the entire point of running the assessment first. We treat the port decision as the output of a performance and porting assessment that profiles the existing path before any migration commitment, because the cheapest port is frequently the one you decide not to do. FAQ When does porting Python inference to C++ or WASM actually pay off? A port pays off when profiling shows Python interpreter and glue overhead is a large share of a latency or throughput budget you are actually missing, or when the target is reduced footprint or client-side (browser/edge) execution that a faster kernel cannot deliver. It does not pay off when model compute or IO dominates the request — in those cases the port moves engineering cost without moving the bottleneck. How do we profile the current Python path before committing to a port? Set an explicit target (e.g. p99 under 40 ms at a given request rate), profile the path under realistic concurrency with a sampling profiler or the PyTorch profiler, and attribute the wall-clock time to model compute, Python overhead, or IO. If the GPU is busy most of the request, compute is the constraint and a port won’t help; if the GPU is starved while Python marshals data, you’ve found a candidate the port can move. What latency or footprint targets justify a port? A port is justified when the gain ceiling — set by the size of the Python-overhead bucket — clears the target you’re missing, or when the target is footprint and cold-start (edge, serverless) or client-side execution. If profiling shows overhead is only a single-digit percentage of a request, the ceiling on improvement is that same percentage, which rarely justifies a full rewrite. What engineering and maintenance cost does a port carry afterward? A ported path adds a second toolchain, a build pipeline to keep green, a smaller pool of engineers who can change it safely, and a divergence risk where the ported path drifts from its Python reference. Every model or schema update has to move both paths together, and the ported one moves slower — a recurring cost no first-day benchmark captures. When does a port move the cost without moving the bottleneck? Whenever the dominant bottleneck is model compute (already-native kernels like cuDNN or FlashAttention) or IO (data loading, preprocessing, network). A port primarily attacks interpreter and glue overhead, so if that bucket is small, the rewrite relocates the same wall-clock time into a more expensive-to-maintain codebase. When does Cython or a partial C-extension close the gap without a full C++/WASM rewrite? When the overhead is concentrated in a hot loop — a tokenizer, a feature transform, a preprocessing step — that you can compile in place while keeping the rest of the system in Python. This captures much of the gain while preserving the Python deployment story and the team’s ability to maintain the code. How do we decide between a WASM/Pyodide path and native C++/CUDA acceleration for a given inference workload? Decide by the target, not by speed: native CUDA wins on raw compute, so choose it when latency or throughput against compute-bound kernels is the goal. Choose WASM/Pyodide when the deployment target is the browser or an edge sandbox and the alternative is a server round-trip you cannot afford or secure — WASM trades compute throughput for portability and isolation. The Decision That Precedes the Port The useful framing is to stop asking “should we port to C++ or WASM?” and start asking “what does our profiling baseline say the bottleneck is, and does a port move it?” One sequence produces a defensible decision with a quantified expected gain; the other produces a faster rewrite of code that was never the constraint. That same discipline applies the moment a ported path is ready to ship — the release-readiness decision for a migrated inference path hinges on whether it actually cleared the target the port was supposed to hit. The honest profiling pass that produces this decision is the port-decision step of our broader GPU and inference optimisation work, and is packaged as the Inference Cost-Cut Pack. The failure class to remember has a simple name: a port that bypasses profiling spends real engineering and leaves the bottleneck exactly where it was.
