An assessment that cannot end in “don’t port” is not an assessment. It is a rewrite that has been given a profiling appendix, and it will move engineering cost without moving the bottleneck.
This matters because of how the work usually gets announced. Someone says “we’re assessing a C++ or WASM port of the inference path,” and by the following sprint two engineers are setting up a CMake tree, a third is looking at Emscripten targets, and the profiling that was supposed to justify the decision has quietly become the paperwork attached to a decision already made. Nobody lied. The scope simply never had a terminating branch that said stop.
So this piece draws a boundary rather than making a performance argument. What the decision work covers, what it deliberately does not, and where the line sits between a bounded pass and a migration commitment.
What does “port-decision work is not a rewrite commitment” mean in practice?
It means the deliverable is a document, not a branch.
A port-decision pass produces three things. A profiling baseline of the existing Python inference path, measured at the batch sizes and concurrency the production path actually sees. An attribution of that measured time across model compute, Python and framework overhead, and IO or data movement. And a cost estimate for the port — engineering weeks plus the ongoing maintenance surface of a second implementation — set against the gain the attribution says is reachable.
None of those artefacts is migrated code. That is the whole point. If the pass ends with 4,000 lines of C++ that “we may as well keep now that it exists,” the decision was never live; sunk cost took the vote before the numbers arrived.
The arithmetic is unforgiving and it is why the boundary has to be structural rather than cultural. If interpreter overhead is 8% of a request that spends most of its budget inside GPU kernels, a perfect language port buys you at most that 8%, minus whatever the new glue costs you back. A pass that has already committed to the rewrite has no incentive to surface that number early — and in our experience it usually surfaces it late, in a retro, after the merge.
The two sides of the line
The most common failure is not disagreement about the target. It is that decision-pass activities and migration activities get scheduled in the same sprint, so the pass inherits the migration’s momentum.
| Belongs to the decision pass | Belongs to a later migration commitment |
|---|---|
| Instrumenting the existing Python path end to end (cProfile / py-spy, CUDA-level timing via Nsight Systems or PyTorch’s profiler) | Standing up a build and packaging pipeline (CMake, Emscripten, cross-compilation) |
| Attributing wall-clock time to model compute, Python overhead, or IO | Reimplementing pre/post-processing in the target language |
| Stating the latency, footprint, or unit-cost target as a number | Pinning CUDA/driver versions for the native path |
| Quantifying the gain a language change can actually reach | Wiring the ported path into CI and release gating |
| Costing the port, including dual maintenance if the Python path survives | Onboarding maintainers for the second implementation |
| Comparing the port against cheaper interventions | Deleting or freezing the Python path |
| Writing the recommendation — including “don’t port” | Monitoring the ported path in production |
Read the right-hand column as a definition by exclusion. Every row there is real work with real value; none of it is decision work, and any of it appearing before the recommendation is signed is a scope breach, not progress.
Warning signs the assessment has already become a rewrite
Four signals show up early enough to act on, and we pay close attention to them when scoping this kind of work:
- There is no stated number. No latency percentile, no image-size budget, no cost-per-thousand-inferences target. Without a target, the pass cannot fail the port, because nothing defines a gap the port has to close.
- The plan has a target language but no baseline. Choosing C++ over Rust over WASM before the profile exists means the deployment surface, not the measurement, is driving. That choice is legitimate work — but it belongs after the “yes”.
- Headcount is allocated past the decision point. If the roadmap already shows port engineers in the two sprints following the assessment, the assessment’s output has been pre-decided. Staff the pass; leave the next sprint unstaffed until it reports.
- Cheaper interventions were never costed. A pass that never priced a Cython annotation on the hot loop, a batching change, a kernel or runtime swap, or a smaller model is comparing the port against nothing.
The clean structural fix is a written termination condition. Name the target, name the attribution threshold that would justify the port — for instance, “Python-layer overhead must own enough of the budget that removing it closes the gap to target” — and agree in advance what happens if the threshold is not met. Time-box the pass so it costs materially less than the port it might avoid; assessments that outgrow the thing they were meant to evaluate stop being a saving.
If the pass returns “don’t port”, what does it hand back?
A “don’t port” outcome is not an empty result, and treating it as one is how the branch gets closed off in the first place. The attribution that killed the port also names the intervention that is left.
- Model compute dominates. The work is on the model or the runtime: precision and quantisation, graph compilation, kernel selection, TensorRT or ONNX Runtime, a distilled architecture. Language is irrelevant here.
- IO and data movement dominate. Look at transfer, serialisation, decode, and batching before anything else. Frame decode in a video pipeline and host-to-device copies both live in this bucket.
- Python overhead is real but narrow. A targeted C-extension or Cython annotation on the hot loop can capture most of the gain without a second full implementation to maintain.
- The target itself is wrong. Sometimes the honest output is that the stated latency number was inherited rather than derived, and the correct next step is to renegotiate it.
Any of those is a better outcome than a rewrite that lands within a few percent of where it started. The avoided cost is the measurable win: inference paths where Python is not the dominant bottleneck get identified before a port is staffed, and the fixed cost of the pass is the price of that information.
Where the port does clear the bar, the rubric for that judgement — expected gain against fully loaded engineering cost — is developed in when porting Python inference to C++ or WASM actually pays off, which is the affirmative counterpart to this boundary. Our broader GPU and inference engineering practice covers the optimisation work that a “don’t port” result usually routes to, and the Inference Cost-Cut Pack is where we run this pass as a bounded engagement whose output is a decision with rationale rather than migrated code.
The uncomfortable part is that this boundary is easiest to hold before anyone has opened an editor and hardest to hold after. Which is why the question worth asking at kickoff is not “how long will the port take?” but: what result would make us walk away, and who is allowed to say it?
Frequently Asked Questions
What does “port-decision work is not a rewrite commitment” mean in practice? A useful Port Decision Work clarification is this. It means the pass ships a document, not a branch. The deliverables are a profiling baseline, a bottleneck attribution across model compute, Python overhead and IO, and a costed comparison of the port against the gain it can reach. If migrated inference code exists when the pass ends, the decision was never genuinely open.
What does a port-decision pass actually deliver, and what does it deliberately leave out? It delivers the baseline, the attribution, the quantified expected gain in the target language, the port’s engineering and maintenance cost estimate, and a recommendation. It leaves out build tooling, packaging, reimplemented pre/post-processing, CI integration, and any production wiring — all of which belong to a separate, later commitment.
How do we structure the assessment so “don’t port” is a real, acceptable outcome? Write the termination condition before starting: a stated latency, footprint or unit-cost target, plus the attribution threshold that would justify the port. Do not staff the following sprint with port engineers, and require that at least one cheaper intervention is costed alongside the port so the comparison has two live options.
Which activities belong to the decision pass and which belong to a later migration commitment? Instrumenting the existing Python path, attributing measured time, stating the target as a number, quantifying reachable gain, and costing dual maintenance belong to the pass. Build pipelines, cross-compilation targets, driver pinning, rewritten glue, CI gating, and production monitoring belong after the port is committed.
What are the warning signs that an assessment has already become a rewrite in progress? No stated numeric target; a chosen target language with no profiling baseline; headcount already allocated past the decision point; and cheaper interventions never costed. Any one of these means the pass has no live “no” branch left.
If the pass returns “don’t port”, what interventions does it hand back instead? The attribution names the alternative. Model-compute-bound paths route to quantisation, graph compilation, or runtime work such as TensorRT or ONNX Runtime; IO-bound paths route to transfer, decode and batching changes; narrow Python overhead routes to a targeted Cython or C-extension fix; and sometimes the honest output is that the target itself needs renegotiating.
How do we scope and time-box the decision pass so it costs less than the port it might avoid? Fix the box against the port estimate, not against curiosity — the pass should be a small fraction of the multi-sprint rewrite it is evaluating. Scope it to the profiling, attribution and costing deliverables only, and treat scope creep into build tooling as the signal that the box has been breached.
Why porting analysis precedes any rewrite
Characterizing overhead before touching production code prevents the expensive mistake of optimizing subsystems that contribute less than five percent of end-to-end latency.
