Before anyone writes a line of C++, the inference path needs a time budget: what share of wall-clock sits in interpreter and framework dispatch, what sits in device compute, what sits in host-device transfer, and what sits in IO wait. That split is the whole decision. You cannot recover more time from a port than the share of wall-clock the port actually touches — so the profile bounds the best case before the migration is priced.

The reflex when a Python serving path misses its latency target is to time the endpoint, see a number that looks too high, and name the interpreter as the cause. End-to-end timing tells you the request is slow. It tells you nothing about why. And the two outcomes it cannot distinguish — compute-bound and dispatch-bound — lead to completely different engineering programmes.

What does profiling a Python inference path mean in practice?

It means producing a per-stage attribution of the request path at the batch sizes and concurrency the service actually runs, in a fixed order, so each measurement constrains the next.

The order matters more than the tooling:

1. Wall-clock the whole request under representative load, at the percentile the business cares about — usually p95 or p99, not the mean. This is the number you are trying to explain.
2. Split host time from device time. A CUDA-aware profiler (Nsight Systems, or PyTorch’s torch.profiler with activities including CUDA) gives you a timeline where kernel execution and host-side Python are separate tracks. If the device track is nearly saturated for the duration of the request, the interpreter is not your problem.
3. Attribute host time. cProfile or py-spy on the serving process, sampled under load rather than on a single synthetic call, separates framework dispatch from your own Python — serialisation, validation, pre/post-processing, ORM calls.
4. Measure transfer separately. Host-to-device and device-to-host copies show as distinct memcpy events on the timeline. Pinned vs pageable memory changes this materially, and it is worth recording which you were using.
5. Isolate IO wait. Any blocking call — object store fetch, upstream HTTP, database read — is time no runtime change will recover. Instrument it explicitly rather than inferring it as residual.
6. Re-run at production batch and concurrency. A single-request benchmark systematically overstates Python’s share, because per-request fixed cost gets amortised as batch grows.

Step 6 is the one most often skipped, and it is the one that flips conclusions. Per-request interpreter and serialisation overhead is roughly constant per request; device compute scales with the batch. Profile at batch 1 and Python looks dominant. Profile at the batch the service actually serves under load and the same code can be 80% device-bound. Both numbers are real; only one of them is relevant to your decision.

Separating interpreter overhead from device compute

The trap is that Python’s asynchronous relationship with the GPU makes naive timing lie. A forward() call in PyTorch enqueues kernels and returns before they finish. Time it with time.perf_counter() and you have measured launch cost, not compute. Time the block after a torch.cuda.synchronize() and you have folded queue drain into your host measurement.

Two practices resolve this in our experience:

• Use CUDA events (torch.cuda.Event with elapsed_time) or the profiler’s device track for compute, never host wall-clock.
• Look at kernel occupancy and gaps on the timeline. Long gaps between kernels with a busy host track is the signature of dispatch-bound execution — many small kernels, each launched from Python. Back-to-back kernels with an idle host is compute-bound.

That second signature is where a port has real headroom, and it is also where CUDA Graphs or graph capture may get you most of the benefit without changing language at all. Worth checking before the rewrite is scoped.

The attribution table that drives the recommendation

Dominant stage (share of p95 wall-clock)	What it means	Lever with the highest expected return	Does a language port help?
Device compute > 60%	The accelerator is doing the work; host is keeping up	Kernel selection, TensorRT or ONNX Runtime graph optimisation, precision (FP16/INT8), batching	Very little — the port moves the remaining minority
Python + framework dispatch > 40% at production batch	Per-request fixed cost dominates; many small ops	Graph capture, operator fusion, batching, then a port of the hot path	Yes — measurable ceiling worth pricing
Host-device transfer > 25%	Data movement, not computation	Pinned memory, transfer/compute overlap on separate streams, keeping tensors on device	Rarely — the copy cost survives the rewrite
IO / upstream wait > 25%	The path is blocked on something else	Async IO, caching, prefetch, fixing the upstream	No — a port is spent on the wrong stage
No stage above ~30%	Cost is diffuse	Reduce stage count; re-profile after each change	Defer the decision

The right-hand column is the one that gets argued about, so state its basis: the shares are read off your own profile, and the ceiling on any port is arithmetic, not judgement. A port that eliminates all interpreter overhead in a path where dispatch is 20% of p95 cannot deliver more than a 20% improvement, and will deliver less once the target runtime’s own dispatch cost is counted.

When the profile says don’t port

This is a legitimate and common outcome, and the profile is the artifact that makes it defensible. If device compute dominates at production batch, the same measurement that killed the port also names the alternative: precision reduction, a better kernel, a graph-compiled execution path, or larger effective batches through continuous batching. We treat the documented avoided cost of a rewrite that would not have moved the target as a real deliverable, not a null result — it is usually the most expensive decision the profile prevents.

Where the profile does support a port, the attribution becomes the input to a separate question: which target runtime. Overhead attribution tells you how much is recoverable; it does not tell you whether C++, Rust, or WebAssembly is the right destination, and it does not price the migration. That work sits downstream, and we develop the cost-model side of it in what a performance and porting assessment engagement actually delivers.

Handing the profile onward

Record the conditions alongside the numbers. A profile without its conditions is an anecdote. Ours carry, at minimum:

• Hardware and driver: GPU model, CUDA and driver versions, container image digest
• Software stack: framework version, TensorRT / ONNX Runtime version, precision mode
• Load shape: batch size, concurrency, request-size distribution, warm vs cold
• Percentiles reported, not just means
• The per-stage shares, each with the tool that produced it

That package is what lets a target estimate be recalculated six months later when the hardware or traffic changes, instead of re-run from scratch. Teams working through this as part of a broader performance programme usually pair it with our GPU engineering work and scope the assessment through services.

One thing profiling does not settle: how much of the measured overhead is structural — inherent to the interpreter — and how much is one badly placed synchronisation call or an accidental CPU-side tensor copy. That distinction changes the port’s ceiling substantially, and it usually takes a second pass on the timeline to answer. Worth doing before the number goes into a cost model.l.

Frequently Asked Questions

Which stages do you measure, and in what order? Measure end-to-end wall-clock at the production percentile first, then split host time from device time, then attribute host time between framework dispatch and your own Python, then transfer, then IO wait. The order is deliberate: each measurement bounds what the next one can be. Finish by re-running the whole sequence at production batch and concurrency.

How do you separate Python and framework dispatch overhead from actual model compute? Never with host-side wall-clock timers, because GPU execution is asynchronous and forward() returns before kernels finish. Use CUDA events or a profiler’s device track for compute, and read the timeline: busy host with gaps between kernels means dispatch-bound, back-to-back kernels with an idle host means compute-bound.

At what batch size and concurrency should the profile be taken? At the ones the service actually runs, plus one step above to see where saturation begins. Single-request benchmarks systematically overstate Python’s share, because per-request fixed cost is constant while device compute scales with batch — a profile at batch 1 can invert the conclusion you would reach at batch 32.

Why does telling IO and transfer wait apart from compute change the recommendation? Because neither survives a language port. Host-device copy cost and blocking upstream calls are properties of the data path, not the interpreter, so time attributed to them is time a rewrite cannot recover. Separating them prevents a port from being scoped against a bottleneck it never touches.

How does the per-stage budget become an upper bound on a port’s gain? It is arithmetic: a port can only recover time spent in the stages it replaces. If interpreter and serialisation overhead is 20% of p95, no port beats a 20% improvement, and the realistic figure is lower once the target runtime’s own dispatch cost is counted. That bound goes straight into the migration cost model and payback window.

What should the profile package contain so the baseline is defensible later? Hardware, driver, framework and runtime versions, container digest, precision mode, load shape (batch, concurrency, request-size distribution, warm/cold), the percentiles reported, and the per-stage shares each tagged with the tool that produced it. With those recorded, estimates can be recalculated when hardware or traffic changes instead of re-measured from zero.

Should you rewrite or optimize in place?

Profiling reveals whether your bottleneck lives in Python overhead or in the operator kernels themselves. Everything else is detail.