A rewrite that lands a real speedup on a stage owning four percent of wall-clock time is still a write-off. That is the shape of the failure: the port worked, and the p99 did not move, because the constraint was never in the ported code.
We see this arrive in a specific way. A team is under latency or cost pressure, someone opens the serving path, sees Python, and the diagnosis writes itself — the language is slow, so rewrite the hot module in C++ or push it onto CUDA. The rewrite ships. The interpreter overhead in that module genuinely drops. And the tail latency the business actually cares about sits within profiling noise of where it started, because the time was going into data loading, serialisation, host-to-device transfer, small-batch dispatch, or a synchronous call to an upstream service that has nothing to do with which language the module is written in.
The port did not fail technically. It moved cost from the runtime bill into the engineering budget and left the constraint untouched.
What “porting moves cost without moving the bottleneck” means in practice
The divergence point is attribution, not ambition. A port aimed at a profiled bottleneck converts engineering effort into throughput. A port aimed at an assumed bottleneck converts engineering effort into a second language in the stack.
The ceiling on any port is the share of wall-clock time the ported stage owned before you started. Nothing in the target runtime changes that arithmetic. If pre-processing and transfer own 70% of a request and the model forward pass owns 25%, a perfect rewrite of the forward pass caps out at a 25% improvement — and only if the rewrite is infinitely fast, which it is not.
That is why “defer the port” is a legitimate outcome of a performance and porting assessment rather than an admission that the assessment wasted its budget. In our experience, a meaningful fraction of the migrations teams arrive already convinced of turn out to be reachable — at the latency target that matters — by batching, caching, async I/O, or overlapping transfer with compute.
How do you tell whether Python is the bottleneck or the data path around it?
You attribute measured wall-clock time to stages, then ask which stages a port would touch. That is a days-long exercise with standard tooling — cProfile or py-spy for interpreter-level attribution, Nsight Systems for host/device timeline overlap, and framework-level tracing in PyTorch to separate kernel time from dispatch and synchronisation. It is not a quarter-long exercise, which is the part teams usually get backwards: they skip the cheap measurement and commit to the expensive rewrite.
The question to hold in mind while reading a profile is uncomfortable and useful: if this stage went to zero, does the number the business cares about move?
Which stages a port actually touches
| Stage | Typical time owner | Does a C++ / CUDA port touch it? |
|---|---|---|
| Request deserialisation, JSON/protobuf decode | Serialisation library, allocation churn | Partly — a port can help if this stage is large, but a serialisation format change is usually cheaper |
| Data loading, decode, disk or object-store fetch | I/O wait | No — port speeds the CPU work, not the wait; async I/O and prefetch do |
| Pre/post-processing in NumPy / OpenCV | Already native under the Python call | Rarely — the heavy loops are compiled already; you are porting the glue |
| Host-to-device transfer | PCIe bandwidth, unpinned memory, no overlap | No — fixed by pinned buffers, batching, and stream overlap |
| Kernel launch / small-batch dispatch overhead | Per-call fixed cost | Sometimes — but graph capture or batching usually reaches it first |
| Model forward pass | GPU compute | No — the kernels are already native; a port changes who calls them |
| Interpreter-level Python loops over elements | CPython bytecode execution | Yes — this is the case where a port is genuinely indicated |
| Synchronous upstream call (auth, feature store, model registry) | Network round trip | No — no target runtime removes a blocking dependency |
Only two rows in that table are honest candidates for a language migration. The rest are the rows where ports go to die, and they are the rows a naive reading of “Python is slow” cannot distinguish from the others. We explore the thresholds and evidence standards that govern the underlying commit-or-defer call in our performance and porting assessment methodology, and the runtime-level profiling practice in more detail on our GPU engineering work.
The recurring cost nobody prices
The write-off is the visible part. The part that shows up later is that a partial port leaves you maintaining a second language, a second build and toolchain, a second set of dependency pins, a second class of crash to debug, and a second on-call surface for the people who now have to reason across the boundary between them. Numerical behaviour can drift across that boundary too — the same activation implemented in two runtimes will not always land on the same side of a decision threshold.
None of that is an argument against porting. It is an argument for knowing what you bought. When the profiled bottleneck really is interpreter-level work in the hot path, a port pays for those recurring costs comfortably and the case is easy to defend to a VP of Engineering. When it is not, you have taken on permanent complexity in exchange for a number inside the error bars.
Recognising the failure after the fact
If the port has already shipped, the diagnostic is short:
- Compare projected against realised speedup. A large gap is not an implementation defect; it is an attribution defect upstream of the implementation.
- Reconstruct what share of the original latency budget the ported stage ever owned. If nobody can answer this from evidence, the rewrite was approved on a suspicion.
- Check whether the current p99 is bounded by the same stage as before. If the ranking of stages is unchanged, the port was orthogonal to the constraint.
- Ask which cheaper fix was tried and measured first. If the answer is none, the batching, caching, and transfer-overlap headroom is still sitting there — and now you also own a second language.
The honest version of this post-mortem often produces a useful second finding: the rewritten module is now the fastest stage in a pipeline still gated by something else, which makes the next optimisation target unusually clear. That is a small consolation for engineer-months, but it is real.
Frequently Asked Questions
What does “porting moves cost without moving the bottleneck” mean in practice?
On Porting Often Moves Cost, the evidence points one way. It means the rewrite succeeded at its stated technical goal and failed at the business goal. Engineering effort was spent making a stage faster that never owned enough of the wall-clock time to matter, so the latency or cost-per-inference number stays put while the migration bill lands on the roadmap. The spend moved from the runtime bill into the engineering budget; the constraint did not move at all.
How do we tell whether Python itself is the bottleneck, or the data path around it?
Attribute measured wall-clock time across stages — deserialisation, data load and I/O wait, pre-processing, host-to-device transfer, kernel dispatch, model compute, upstream calls — using py-spy or cProfile alongside a device-side timeline from Nsight Systems. Python is the bottleneck when interpreter-level loops over individual elements dominate. When compiled libraries, transfers, or network waits dominate, the interpreter is a bystander.
Which stages does a port actually touch — and which ones does it leave untouched?
A port reliably helps interpreter-level element-wise work and, sometimes, per-call dispatch overhead. It does not touch I/O wait, PCIe transfer time, blocking upstream network calls, or GPU kernel time that was already executing native code. NumPy and OpenCV pre-processing is a common trap: the heavy loops there are compiled already, so a rewrite ports the glue rather than the work.
What profiling evidence should exist before a rewrite is approved?
A profiled baseline on representative inputs, not synthetic ones; a stage-level breakdown of the latency budget with a percentage attached to each stage; an explicit ceiling calculation showing the maximum speedup available if the ported stage went to zero; and a comparison of that ceiling against the target the business actually needs to hit. If those four items do not exist in writing, the rewrite is being approved on a suspicion.
Which cheaper fixes should be exhausted first?
Batching (to amortise per-call dispatch cost), pinned memory with stream overlap (to hide host-to-device transfer behind compute), async or prefetched I/O (to remove serial wait), caching of repeated upstream lookups, and a cheaper serialisation format. Each is typically days of work against a port’s engineer-months, and each targets a stage a language migration would not have touched.
What are the recurring costs of a partial port?
A second language in the stack means a second build and toolchain, a second dependency surface, a second class of failure mode at the boundary between runtimes, and a smaller pool of engineers who can safely change that code. Numerical differences across the boundary can also shift model behaviour subtly. These costs recur every quarter; the speedup, if it was real, was paid once.
How do we recognise, after the fact, that a completed port failed to move the constraint?
Compare projected against realised speedup, and reconstruct what fraction of the original latency budget the ported stage owned. If the stage ranking in the profile is unchanged and the p99 is still gated by the same thing as before the migration, the port was orthogonal to the constraint — an attribution failure that happened before a line of the rewrite was written.
If a migration is being scoped in your organisation right now, the question worth asking in the next planning meeting is not which target runtime to choose. It is: what percentage of our current latency budget does the code we intend to rewrite actually own, and who measured it?
When cheaper compute still leaves you waiting
Halving your instance cost feels like progress until you realize network I/O or preprocessing still gate every request at the same 200 ms floor. If Porting Often Moves Cost is on your roadmap, the next step is to map it onto your own constraints rather than copy a reference architecture.
