A port is priced in engineering weeks. The evidence that justifies it costs a few days of instrumentation, and it has to exist before the rewrite, not after. The profiling pass does not decide the port — it produces the number the decision needs: what fraction of measured wall time actually sits in the Python layer, under the batch sizes and concurrency the production path really sees.

That number is a ceiling. If Python interpreter overhead accounts for 12% of request wall time, a flawless rewrite in C++ buys at most 12% — and only if the new implementation introduces no cost of its own. Everything else in the port decision is negotiation about how much of that ceiling is worth paying for.

The common failure is not sloppy measurement. It is measurement at the wrong granularity: a timer wrapped around model(x), compared against the target, with the gap attributed to “Python being slow” because no other candidate was instrumented.

What does profiling a Python inference path actually involve?

It involves treating the request, not the model call, as the unit of measurement. A production inference request typically passes through six distinct stages, and each has a different response to a language change:

1. Request handling — HTTP/gRPC deserialisation, validation, auth, queue wait.
2. Preprocessing — decode, resize, tokenise, normalise; often pure Python or NumPy.
3. Host-to-device transfer — tensor marshalling, pinned vs pageable memory, cudaMemcpy.
4. Kernel execution — the model forward pass on the GPU.
5. Postprocessing — NMS, detokenisation, thresholding, format conversion.
6. Serialisation and response — JSON/protobuf encoding, network write.

Instrument all six or the attribution is guesswork. In our experience the two most frequently un-instrumented stages are queue wait and host-to-device transfer, and they are also the two most likely to surprise a team that believed the interpreter was the problem.

Granularity matters more than tool sophistication. Per-stage timers around the six boundaries above, recorded per request and aggregated to p50/p95, will tell you more than a flame graph averaged over a warm loop. Use the flame graph second, to explain a stage that the timers show is expensive.

The three-bucket attribution

Every millisecond you measure has to land in exactly one of three buckets. This is the analytical core of the pass, and it is where most profiling exercises stop short.

Bucket	What belongs in it	How you recognise it	Does a port move it?
Model compute	GPU kernel time, cuBLAS/cuDNN calls, attention kernels, CPU-side BLAS when running on CPU	CUDA events or NSight Systems timeline shows contiguous kernel occupancy; the gap closes when you shrink the model, not the glue	No. Kernel time is language-independent.
Python / framework overhead	Interpreter dispatch, per-call framework bookkeeping, GIL contention, Python-level loops in pre/postprocessing, object churn	py-spy sampling lands in Python frames; latency grows with call count more than with tensor size; scales badly with concurrency	Yes, this is the addressable fraction.
IO and data movement	Disk reads, object-store fetches, network round trips to other services, host↔device copies, queue wait	Wall time far exceeds CPU time; strace/network traces or async-wait spans dominate; unaffected by CPU pinning	No, not by a language change.

Two attribution traps recur. First, CUDA is asynchronous: a naïve timer around a launch measures the launch, not the execution, so kernel time gets misfiled as Python overhead. Synchronise explicitly with torch.cuda.synchronize() or use CUDA events before trusting any per-stage split. Second, tensor-transfer time hides inside what looks like model compute unless the copy is timed separately — pageable-memory copies on a busy host are a real and separately fixable cost.

Tooling that produces a trustworthy split

No single profiler covers all three buckets. The combination we reach for:

• Per-stage timers (time.perf_counter, emitted as structured log fields or spans) — the spine of the pass. Cheap enough to leave on in staging, and the only source of a true p95.
• py-spy — sampling profiler, attaches to a live process without restart, and shows GIL contention across threads. This is the right tool for “is the interpreter the problem” because it works under real load.
• cProfile — deterministic call counts for a single-request path. Useful for finding a Python-level hot loop; its per-call overhead makes it unsuitable for latency numbers.
• PyTorch Profiler / NSight Systems — the only reliable source of GPU kernel time and the CPU↔GPU timeline. NSight Systems shows launch gaps, which is how you distinguish a GPU-bound path from a path that is starving the GPU.
• OpenTelemetry spans — if the path crosses services, IO wait is only visible in a distributed trace.

Numbers from these tools are project-specific operational measurements, not portable benchmarks; the split you find on your path says nothing about anyone else’s.

Profile under load, or you have measured the wrong system

A single-request microbenchmark systematically flatters the interpreter’s role and hides everything that only appears under concurrency. Three conditions have to be reproduced:

• Representative batch size. Per-call Python overhead is amortised across a batch. At batch 1 the interpreter may be 30% of the path; at batch 32 the same absolute overhead is a few percent, and the GPU is now the constraint. Profile at the batch size the serving layer actually forms.
• Target concurrency. GIL contention, thread-pool saturation, and queue wait are emergent properties of concurrent load. A path that looks clean at one request in flight can spend most of its p95 waiting on the GIL at sixteen.
• Warm and cold state. Separate the first-request path (weight load, CUDA context creation, graph compilation, JIT warm-up) from steady state. Mixing them produces a p99 that no optimisation will explain.

Report p50 and p95 per stage, plus sustained throughput at target concurrency. A mean is not decision-grade: ports are usually justified against a tail-latency SLO, and the bucket that dominates p95 is frequently not the bucket that dominates p50.

Turning the split into a decision input

The arithmetic is deliberately blunt. Take the Python/framework bucket’s share of p95 wall time, subtract the fraction of it you cannot eliminate (framework bookkeeping that a native runtime still has to do, and the serialisation the interface requires), and you have the realistic upper bound on the port’s gain. Compare that bound with the gap between current p95 and the target.

Three outcomes:

• Bound comfortably exceeds the gap — the port is a live option, and the next question is which target language and what the ongoing cost looks like. We work through that trade-off in when porting Python inference to C++ or WASM actually pays off, which takes the profiled split as its input.
• Bound is smaller than the gap — a port cannot reach the target on its own. The path forward is model-side (quantisation, distillation, a smaller encoder), batching and scheduling changes, or IO restructuring.
• Bound is a few percent — the bottleneck sits outside the Python layer. Do not fund the port. Kernel-dominated and IO-dominated paths are the two clearest cases: 90% of wall time in GPU kernels or in a remote object-store fetch will not shift because the host glue changed language.

That last outcome is the profiling pass earning its cost. The engineering saved by not rewriting a path whose time lives in cudaLaunchKernel and S3 latency is the most reliable return the pass produces, and it is invisible unless someone measures first. This profiling step is the first stage of our GPU and inference engineering work, and it feeds the port-decision step of the Inference Cost-Cut Pack.

The baseline is also your post-port evidence

One practical reason to keep the instrumentation rather than delete it after the decision: the same per-stage p50/p95 record is the before-state a ported path has to be measured against. Without it, the post-port benchmark has nothing to compare to except a remembered number from a different load profile, and “the port made it faster” becomes unfalsifiable.

Keep the harness, the load profile, and the stage boundaries under version control alongside the code. A profiling baseline that cannot be re-run in six months is an anecdote.

Frequently Asked Questions

Which stages of the inference path do we instrument, and at what granularity?

Instrument six boundaries: request handling and queue wait, preprocessing, host-to-device transfer, kernel execution, postprocessing, and response serialisation. Per-request timers at those six boundaries, aggregated to p50 and p95, are the right default granularity. Go finer only inside a stage the timers have already shown to be expensive.

How do we attribute measured wall time to model compute versus Python overhead versus IO?

Assign every measured millisecond to exactly one of three buckets — model compute, Python/framework overhead, IO and data movement — using CUDA events or NSight Systems for kernel time, py-spy for interpreter and GIL time, and traces or wall-minus-CPU time for IO. Synchronise CUDA explicitly before trusting any split, because asynchronous launches otherwise misfile kernel time as Python overhead.

What tooling combination gives a trustworthy picture?

Per-stage perf_counter timers form the spine and are the only source of a real p95. py-spy answers the interpreter question under live load, cProfile finds Python-level hot loops in single-request runs, and the PyTorch Profiler or NSight Systems supplies GPU kernel time and launch gaps. Add distributed tracing if the path crosses service boundaries.

How do we profile under realistic batch size, concurrency, and GIL contention?

Reproduce the batch size the serving layer actually forms, the concurrency the SLO is written against, and steady-state rather than cold-start conditions — reported separately. Per-call Python overhead amortises across batches and GIL contention only emerges under concurrency, so a single-request microbenchmark reliably overstates the interpreter’s share.

How do we turn the attributed split into a ceiling on the gain a port could deliver?

Take the Python/framework bucket’s share of p95, subtract the portion any native implementation would still pay (framework bookkeeping, interface serialisation), and treat the remainder as the maximum achievable gain. If that ceiling is smaller than the gap between current p95 and the target, no language change reaches the target and the work belongs elsewhere.

What profiling results indicate a port should not be funded?

A path where GPU kernels or remote IO own the large majority of p95, and where the Python bucket is in the low single digits, will not get faster in C++ or WASM. The same conclusion holds when latency scales with tensor size rather than call count, which points at compute and memory bandwidth rather than interpreter dispatch.

If your current attribution is an opinion rather than a measurement, what would change in the port decision once you had the p95 split in front of you?

Three profiling results that change the porting decision

Most porting regrets trace back to profiling the wrong layer or skipping concurrency effects entirely. Profile Python Inference Path rewards teams that measure first and argue later — start with the smallest instrumented slice and let the numbers settle the design.