“This path is computationally expensive.” You have heard that sentence in a standup, and it usually arrives with an implied fix attached: rewrite it in something faster. The instinct is understandable and, more often than not, wrong in a way that costs weeks. “Computationally expensive” is not a diagnosis. It is a symptom label that gets pinned to any inference path that feels slow — and until you know which section of the path is expensive and against which budget, the label tells you nothing about the fix. The expert reading is narrower and more useful. Expensive against what — latency, throughput, or memory footprint? And expensive where — in the Python glue that orchestrates the path, in the model compute the GPU actually runs, or in the IO that feeds it? A hot loop dominated by Python interpreter overhead is expensive in a way a targeted Cython annotation can close. A kernel saturating the GPU is expensive in a way no amount of static typing will touch. These are not the same problem, and treating them as one is how a team ends up costing a full C++/WASM port for something a C-extension would have fixed in an afternoon. The catch-all label hides three different costs When people say a path is computationally expensive, they are almost always pointing at one of three distinct cost centres — and the word “computationally” makes only one of them sound like the culprit. The first is glue-code cost: the Python interpreter overhead of the orchestration around the model. Loops iterating in pure Python, per-element type dispatch, attribute lookups, object allocation churn, serialisation between stages. This cost is real and can dominate wall-clock time, but it is not “compute” in the sense of arithmetic throughput. It is the cost of a dynamic interpreter doing bookkeeping. The second is model compute: the matrix multiplies, convolutions, and attention kernels that constitute the actual model. On a GPU this is where FLOPs are spent. If a kernel is already saturating the device, this cost is genuinely computational — and it is bounded by the hardware, not by how you wrote the surrounding Python. The third is IO and data movement: reading inputs from disk or network, host-to-device transfers over PCIe, decoding media, deserialising tensors. A path can spend most of its latency here while every profiler that only looks at kernel time reports the GPU as underused. The reason this distinction matters is that each cost centre responds to a completely different intervention. Confusing them is the divergence point that separates a cheap fix from an expensive one. Expensive against what budget? Before attribution, name the budget. The same profiled path can be “fine” or “unacceptable” depending on which of three references you are measuring against, and the reference changes the answer. Latency budget — the time for a single request end to end. If you have a 50 ms per-frame budget for a real-time overlay, a 12 ms Python preprocessing step is expensive; the same step is invisible in a batch job. Throughput budget — requests or tokens per second across the deployment. Here per-request Python overhead that does not batch amortises poorly and becomes the dominant cost even when each individual call looks cheap. Footprint budget — memory and the hardware you are willing to provision. A path that hits latency and throughput targets but only on a device you cannot afford is expensive against footprint. A section can be under one budget and blowing another. Interpreter overhead that is negligible in a latency-relaxed batch pipeline can be the throughput killer under load, because it does not amortise across a batch the way a fused GPU kernel does. State the budget first, then attribute — otherwise “expensive” is just a feeling. How do you tell Python overhead from model compute? This is the question that decides everything downstream, and it is answerable with measurement rather than intuition. The pattern we use, and the one behind our writeup on profiling the Python inference path before a C++ or WASM port, is to separate wall-clock time into interpreter time and device time before proposing any fix. A CPU sampling profiler (py-spy, cProfile, or scalene) tells you how much wall-clock time is spent in the Python interpreter itself versus in calls that hand off to native code. If a hot function shows most of its self-time in Python frames — loops, comprehensions, per-element calls — that is interpreter overhead. A GPU profiler (Nsight Systems, or the PyTorch profiler with CUDA activity enabled) tells you how much time the device is actually busy running kernels. Line these two views up on the same timeline and the picture usually resolves quickly. The tells, in our experience across optimization engagements (observed pattern, not a benchmarked constant): Python-expensive: the GPU timeline is full of gaps, kernels are short and frequent, and the CPU profiler shows self-time concentrated in interpreter frames. The device is waiting on Python. Compute-expensive: the GPU timeline is dense, a few kernels dominate, and the CPU profiler shows most time in native _C calls or CUDA sync. Python is not the constraint; the arithmetic is. IO-expensive: both the CPU-Python time and the GPU-busy time are low relative to wall clock — the path is blocked on transfers, decode, or fetches. Our note on reading FFmpeg encode/decode timings before you port an inference path walks through this case where media decode, not the model, owns the latency. Cost attribution decision table The whole point of getting the definition right is to make the Cython-vs-rewrite decision quantifiable. Here is the mapping from where the cost lives to what actually reduces it. Dominant cost How it shows up in profiling What reduces it What does not Python interpreter overhead (glue code) High interpreter self-time, sparse GPU timeline, many short kernels Cython C-extension, vectorisation into NumPy/Torch, removing per-element Python loops GPU upgrade; more precision tuning Model compute (kernel-bound) Dense GPU timeline, few dominant kernels, high device utilisation Kernel fusion, TensorRT/torch.compile, quantisation, a native CUDA path Cython on the surrounding Python — the interpreter is not the constraint IO / data movement Low CPU-Python and low GPU-busy time vs wall clock; PCIe or fetch waits Async prefetch, batching transfers, pinned memory, decode offload Rewriting compute code; the compute is already idle The table is the argument in one screen: a Cython extension is a scalpel for glue-code cost and does nothing for the other two rows. This is why the GCC compiler flags for Cython extensions matter only once you have confirmed the cost is actually living in the interpreter — otherwise you are tuning -O2 and -march on code that was never the bottleneck. What Cython can and cannot touch A Cython C-extension replaces interpreted Python bytecode with compiled C for the annotated sections. That directly attacks interpreter overhead: type-annotated loops stop paying per-iteration dispatch cost, object churn drops, and hot numeric sections run at C speed. When a path is Python-expensive, a well-placed Cython annotation on the hot loop can recover most of that section’s cost for a fraction of the effort a full port would take (observed across engagements; not a published benchmark). What Cython cannot touch is model compute already running as native kernels. If your matrix multiply is a cuBLAS call, wrapping the Python that invokes it in Cython changes nothing — the arithmetic was never interpreted. Cython also does not fix IO: a path blocked on PCIe transfers or media decode stays blocked whether the surrounding orchestration is interpreted or compiled. This is the trap the CCU’s urgency framing names directly. A team that reads “computationally expensive” as “needs a rewrite” costs a full C++/WASM port for a problem where the addressable fraction — the part a C-extension could reach — was small and the rest lived in kernels or IO the port would not improve either. Understanding how the work divides across CPU, GPU, and other targets is a prerequisite here; our overview of heterogeneous architecture for inference across CPU, GPU, and WASM targets covers how those boundaries determine what any single-language rewrite can and cannot reach. When does compute-expensive actually justify a native path? Sometimes the cost really is in the compute, and a native C++/CUDA path is the right intervention. The signal is specific: the GPU timeline is dense, a handful of kernels dominate, device utilisation is high, and the interpreter overhead is already small relative to total time. At that point Cython has nothing left to give — the interpreter is not on the critical path — and the levers that remain are kernel-level. Reading peak versus achieved throughput on an inference path helps here: if achieved throughput is close to the device’s realistic ceiling, the fix is algorithmic or precision-level, not a language swap in the glue. The distinction between “port the glue” and “port the compute” is decision-grade only when it rests on numbers. Before authorising either, the profiling pass should produce three figures: the profiled share of latency attributable to interpreter overhead versus model compute versus IO, the fraction of the path a C-extension could actually address, and the estimated engineering hours for each option. Those numbers are what turn “is this actually compute-expensive, or just Python-expensive?” from a hunch into a decision. This vocabulary is exactly the input our Inference Cost-Cut Pack profiling pass uses to attribute cost before the Cython-vs-rewrite branch is chosen, and the performance-assessment methodology defines how that attribution is measured rather than guessed. FAQ What matters most about computationally expensive in practice? “Computationally expensive” is a symptom label pinned to any inference path that feels slow. In practice it hides three different costs: Python interpreter overhead in the glue code, model compute in the GPU kernels, and IO or data movement. The phrase only becomes actionable once you attribute the cost to one of those sections and name the budget you are measuring against. Expensive against what — latency, throughput, or footprint budget — and why does the reference budget change the answer? The same profiled section can be fine or unacceptable depending on the reference. A Python step that is invisible in a latency-relaxed batch job can be the dominant throughput killer under load, because per-request interpreter overhead does not amortise across a batch. Name the budget — latency, throughput, or memory footprint — before attributing cost, or “expensive” is just a feeling. How do you tell whether a hot section is expensive because of Python interpreter overhead or because of the model compute itself? Separate wall-clock time into interpreter time and device time. A CPU sampling profiler shows how much self-time sits in Python frames; a GPU profiler shows how busy the device actually is. Python-expensive paths show sparse GPU timelines with interpreter self-time concentrated in loops; compute-expensive paths show dense GPU timelines dominated by a few kernels. Which kinds of computationally expensive cost can a Cython C-extension actually reduce, and which does it leave untouched? Cython replaces interpreted Python with compiled C, so it directly attacks glue-code interpreter overhead — hot loops, per-element dispatch, object churn. It does nothing for model compute that already runs as native cuBLAS or cuDNN kernels, because that arithmetic was never interpreted. It also does not fix IO-bound paths blocked on PCIe transfers or media decode. How do you measure the cost of a section of an inference path so ‘expensive’ becomes a number you can act on? Run a paired profiling pass: a CPU profiler for interpreter self-time and a GPU profiler for device-busy time, lined up on the same timeline. Produce three figures — the share of latency from interpreter overhead versus model compute versus IO, the fraction addressable by a C-extension, and the estimated hours per fix. Those numbers turn the question into a decision. When does a section being genuinely compute-expensive signal that a native C++/CUDA path — not Cython — is the right intervention? When the GPU timeline is dense, a handful of kernels dominate, device utilisation is high, and interpreter overhead is already small relative to total time. At that point Cython has nothing left to give because the interpreter is not on the critical path, and the remaining levers are kernel-level: fusion, precision, or a native CUDA rewrite. The next time “computationally expensive” surfaces in a review, treat it as an open question, not a verdict. Ask which section and which budget — and require the profiler split before anyone costs a port. The expensive mistake is not the slow path; it is naming the fix before you have attributed the cost.
