Set the thresholds before the profiler runs. A porting decision is a capital allocation decision: four numbers — target latency at the percentile that matters, footprint against the deployment envelope, unit cost per inference, and the engineering-months the migration will consume — decide whether a rewrite earns its cost or should be deferred. Choose those numbers first and the profiling run becomes a test with a pass/fail line. Choose them afterwards and the profile becomes a narrative you fit around a decision you already made.
The failure is ordinary. A serving path feels slow, someone names Python as the cause, and a C++ or CUDA port lands on next quarter’s roadmap without anyone stating what latency the business actually needs. Six engineering-months later the p99 has moved eleven percent, because the time was never in the interpreter.
Which thresholds justify committing to a port?
Four thresholds, and the port has to clear the ones that are binding — not all four, and not the ones that are merely interesting.
Latency at a named percentile. “Faster” is not a threshold. “p99 under 120 ms for the checkout scoring call, measured at production concurrency” is. Pick the percentile the product contract or the upstream timeout actually cares about. Mean latency almost never is it; a port that halves the mean and leaves a serialisation stall in the tail has bought nothing.
Footprint against the deployment envelope. Memory ceiling on the target instance, or binary size where the artifact ships to a device or a browser. This threshold is often the one that makes a port unavoidable rather than merely attractive — a Python runtime plus PyTorch cannot be squeezed into a WebAssembly bundle, so if the deployment surface is the browser, footprint decides before latency gets a vote.
Unit cost. Cost per inference, per request, or per processed frame, at the traffic level you actually run. Unit cost is the threshold that converts a performance argument into a finance argument, because it multiplies by volume.
Engineering-months. The migration estimate, built from the code you have — how much of the hot path is genuinely portable, how many library dependencies have no equivalent in the target runtime, how much test coverage exists to prove behavioural parity afterwards. This is the number teams skip, and skipping it is why payback windows go unstated.
Payback falls out of the last two: the unit-cost delta times projected volume, against the engineering-months priced at your loaded rate. A migration that pays back inside two or three quarters is straightforwardly defensible. One that pays back in three years is a bet on your traffic forecast, not on your profiler.
The rubric
| Signal from the profiled baseline | Verdict | Why |
|---|---|---|
| Bottleneck is compute in code the port would replace, and it holds >50% of wall-clock time | Commit — modelled speedup is bounded by that share and still clears the target | The port touches the dominant cost |
| Footprint or deployment surface makes the current runtime impossible (browser, embedded, edge device) | Commit — footprint is a hard gate, not a trade-off | No amount of tuning ships a Python runtime into WASM |
| Unit-cost delta × projected volume pays back the engineering-months inside ~2–3 quarters | Commit — with the payback estimate stated in the recommendation | Finance case stands on its own |
| Bottleneck is I/O, host-to-device transfer, batching policy, or an upstream synchronous call | Defer — fix the stage that owns the time first | A rewrite of the wrong stage moves cost, not the metric |
Compiler or runtime headroom untested (baseline built at -O0, no TensorRT/ONNX Runtime path tried, no batching change attempted) |
Defer — exhaust the cheap headroom, then re-baseline | Cheap wins reorder the entire case |
| Modelled speedup clears the target but only at the mean, not at the binding percentile | Defer — the target is not actually met | Tail cost lives elsewhere |
| No stated threshold exists for latency, footprint, or unit cost | Cannot decide — set the thresholds, then profile | There is nothing for the evidence to be tested against |
The row that matters most in practice is the fourth one. When profiling shows the bottleneck sits outside the code you were planning to rewrite, the honest answer is defer, and the rubric is what lets you say so with evidence rather than instinct. We see this pattern regularly on inference paths where the accelerator is idle most of the request window and nobody has looked at the batching policy. The attribution work that produces this finding is covered in profiling a Python inference path before deciding on a port; the rubric here is what you read that attribution against.
What a defensible defer looks like
A defer is not a refusal to decide. It is a decision with a stated trigger for revisiting.
A defensible defer names four things: the threshold that was not met, the stage that actually owns the time budget, the cheaper intervention to try instead, and the condition under which the port comes back on the table. That last item is what stops the defer from becoming permanent drift — for example, “re-assess if p99 traffic doubles” or “re-assess after the batching change lands and we re-baseline.” Amdahl’s law does the bookkeeping: if the portable share of wall-clock time is 30%, no target runtime — C++, Rust, CUDA, or a hand-vectorised kernel — can deliver more than a 1.4× end-to-end improvement, however good the rewritten code is. That ceiling is the single most useful number to put in front of a stakeholder who wants the port for other reasons.
The relationship between the profiled baseline, the ranked target options, and this defer-or-commit layer is the whole point of a performance and porting assessment — the measurement produces the evidence, the rubric produces the recommendation, and the two stay separable so a later stakeholder can re-check the reasoning without re-running the profiler. Where the port does clear the bar, the GPU engineering practice owns the migration itself.
How the rubric shifts on constrained runtimes
Server-side GPU work and constrained-runtime work weigh the four thresholds differently, and treating them the same is a common source of bad ports.
| Server-side GPU target | Constrained runtime (mobile, WASM/WebGL) | |
|---|---|---|
| Binding threshold, usually | Unit cost per inference | Footprint and cold-start time |
| Latency framing | p95/p99 under concurrency | Single-request latency on the worst supported device |
| Fallback if the port fails | Scale horizontally, absorb cost | None — the feature does not ship |
| Payback logic | Cost delta × volume | Feature enablement, not cost saving |
| Portable share of runtime | Often small; compute already on-device | Often large; the whole path moves |
On a constrained target the payback calculation frequently does not apply at all, because there is no cheaper alternative to compare against. The port is the cost of the feature existing. That makes the decision simpler but the estimate harder: engineering-months on a WASM or WebGL target carry more uncertainty than the same work in C++, because the ecosystem of equivalent kernels and numerical primitives is thinner and behavioural parity takes longer to prove.
Where this leaves the harder question: the rubric tells you whether a port clears a threshold, but it cannot tell you how confident to be in the engineering-months estimate that anchors the payback window. That estimate is the softest number in the chain, and it is the one worth stress-testing hardest before anyone signs.
Frequently Asked Questions
What does “porting earns its engineering cost” mean in practice — which artefacts and numbers decide it? A common Porting Earns Engineering Cost question is worth clarifying. It means the modelled gain on a named target metric, bounded by the share of wall-clock time the port can actually touch, is large enough to repay the migration estimate inside an acceptable window. The deciding artefacts are a profiled baseline on representative inputs, per-target gain estimates with their measurement basis stated, and a migration cost model built from your actual code structure — not a general opinion about language speed.
How do we estimate migration cost in engineering-months well enough to compare it against the modelled gain? Build it from the code, not from analogy: count the hot-path modules that must be reimplemented, the third-party dependencies with no equivalent in the target runtime, and the test coverage you would need to prove behavioural parity. Estimate as a range rather than a point, and state which assumption dominates the uncertainty — usually parity testing on numerics, which is where WASM and WebGL targets in particular tend to overrun.
What profiling findings should stop a port outright? Any finding that puts the dominant cost outside the code the port would replace — I/O wait, host-to-device transfer, serialisation, small-batch dispatch, or a synchronous upstream call. Also stop when the baseline itself is untrustworthy, for example built without optimisation flags or without trying the available runtime paths, because that inflates the apparent gain from the rewrite.
How does the rubric change when the target is a constrained runtime rather than a server-side GPU? Footprint and cold-start displace unit cost as the binding threshold, and payback logic largely disappears because there is no cheaper alternative to compare against — the port is what makes the feature possible at all. In exchange, the engineering-months estimate carries more variance, so the decision hinges on estimate confidence rather than on the size of the modelled speedup.
ROI thresholds that justify a port
Porting makes financial sense when recurring inference costs exceed six months of engineering effort—otherwise, tune what you have. Everything else is detail.
