Inference is the phase where a trained model serves live predictions. Every time a user submits a query, uploads an image, or triggers an autocomplete, the model runs forward once and returns an answer — and that single forward pass costs money that recurs for the life of the feature. Training the model was a one-time project expense. Inference is the meter that never stops running. That distinction sounds obvious once stated, but it quietly breaks the financial model of a lot of AI features. The naive view treats inference as a free byproduct of having done the hard work of training. You built the model; now it just answers things. The cost shows up later as an undifferentiated line on a cloud bill, lumped in with everything else, attributable to nothing in particular. By the time someone asks “why is this feature eating margin?”, the answer is buried in GPU-seconds nobody was counting. The expert view starts somewhere else. Inference is the recurring cost centre of production AI. Each request consumes a measurable quantity of compute — tokens generated, GPU-seconds occupied, or third-party API calls billed — and those quantities aggregate into a cost-per-request that scales directly with usage. Teams who hold that framing can set unit-economics targets before a feature ships. Teams who conflate inference with training spend end up optimising the wrong line item, trimming a fixed cost while the variable one grows underneath them. What Is Inference in AI? Inference is what a model does after training is finished: it takes an input it has never seen and produces an output using the parameters fixed during training. No weights change. No gradients are computed. The model is read-only, and the only work is the forward pass — the sequence of matrix multiplications and activations that turn an input tensor into a prediction. For a large language model, one inference call typically means: encode the prompt, then generate output tokens one at a time, each token requiring a full forward pass through the network conditioned on everything generated so far. For a vision model, it might be a single forward pass over an image to produce a classification or a set of bounding boxes. The mechanics differ by architecture, but the economic shape is the same — compute is consumed per request, and the meter resets to zero only when no one is using the feature. This is the conceptual foundation that cost-per-request as the right production-AI optimisation target rests on. You cannot meaningfully target a cost-per-request number until you accept that inference is a per-request cost event. This primer establishes that; the cost-per-request argument builds on it. How Does Inference Differ From Training in an AI Workload? The two phases share a model architecture and almost nothing else economically. Training is a capital-like event: you spend a large amount of compute once (or periodically, when you retrain) to produce a set of weights. Inference is an operating expense: you spend a small amount of compute every single time the feature is used, indefinitely. This is why the common question “how much did the model cost?” is under-specified. It conflates two costs with completely different behaviour over time. Inference vs Training: How the Costs Behave Dimension Training Inference When it happens Once, or on a retrain cadence Every request, for the life of the feature Cost shape Large fixed/capital-like spend Small recurring variable spend Scales with Dataset size, model size, epochs Request volume, tokens per request, latency target Compute pattern Forward + backward pass, gradient updates Forward pass only, weights frozen What you optimise Time-to-train, convergence cost Cost-per-request, p95 latency, throughput Who notices it The team, during the project The CFO, after the feature scales The practical consequence: a model that was cheap to train can be ruinous to serve, and a model that was expensive to train can be perfectly economical in production if its inference footprint is small. Optimising the wrong one is the default failure mode of teams who treat the two as a single “model cost”. Why Does Every Inference Request Carry a Recurring Cost? Because every request does real, billable work. There is no caching trick that makes a genuinely novel request free — a new prompt, a new image, a new user context all require the model to actually run. Even when caching helps (identical prompts, key-value cache reuse within a generation), the marginal request still occupies hardware for a measurable interval. What gets consumed depends on where and how the model runs: Tokens — for LLM serving, both input (prompt) tokens and output (generated) tokens. Output tokens are usually the dominant cost because each one requires its own forward pass. GPU-seconds — for self-hosted models, the wall-clock time a request occupies an accelerator, multiplied by the hourly cost of that accelerator. A request that holds an A100 or H100 for 400 milliseconds has a directly computable cost. API calls — for managed model endpoints (OpenAI, Anthropic, Bedrock, Vertex), a per-call or per-token charge billed by the provider, where the same per-request logic applies but the meter is someone else’s. The precision you serve at moves these numbers directly. Running a model in FP16 versus INT8, or applying quantization to shrink the memory footprint, changes both throughput and cost-per-request — a trade-off that precision as an economic lever in inference systems treats as a first-class measurement concern rather than an afterthought. The point for this primer is narrower: precision is one of several knobs that exist because inference is a metered per-request event, not a fixed cost. What Consumes Resources During an Inference Call? A useful way to see this is to trace a single LLM request through a self-hosted serving path. The runtime — whether TensorRT-LLM, vLLM, or a Triton Inference Server deployment — receives the prompt, tokenises it, runs the prefill stage over all prompt tokens at once, then enters the decode loop, generating output tokens one at a time. Each decode step reads the model weights from HBM, attends over the growing key-value cache, and emits one token. The cost drivers fall out of that trace directly: Prompt length sets the prefill cost and the size of the KV cache that every subsequent decode step must attend over. Output length sets the number of decode steps, and decode is where most LLM inference time goes. Model size sets how much weight data moves from HBM per step — large models are often memory-bandwidth-bound, not compute-bound, during decode. Batch size determines how many requests share a single forward pass, which is the primary lever for amortising fixed per-step overhead across more requests. This is also where the latency-versus-throughput tension lives. Batching more requests together raises throughput and lowers cost-per-request, but each individual request waits longer — a trade-off the throughput-vs-latency relationship in AI inference examines as a measurement discipline. For our purposes: the same forward pass that costs money also has a latency budget, and you cannot tune one without watching the other. Where Does Inference Run in a Deployed AI Serving Path? Inference runs wherever the serving path puts it, and the placement decision shapes the cost structure. The common deployment patterns: Managed API endpoint — you send requests to a provider’s hosted model and pay per token or per call. Lowest operational burden, no GPU to manage, but the unit cost is set by the provider and your margin is whatever you can charge above it. Self-hosted on cloud GPUs — you run the model on rented accelerators (cloud GPU instances) behind your own serving stack. You own the GPU-seconds and the cost-per-request math, including the cost of idle capacity when traffic is low. Self-hosted on owned hardware — you run on hardware you bought or colocated. Highest fixed cost, lowest marginal cost at sustained high utilisation. Edge / on-device — the model runs on the user’s device. Marginal serving cost approaches zero for you, but the model must fit the device’s memory and latency constraints. Each pattern turns inference cost into a different financial object — a variable per-token charge, a GPU-second rate against a utilisation curve, or a fixed amortised capital cost. None of them make inference free; they relocate where the meter sits and who reads it. How Is Inference Cost Per Token Calculated for an LLM? Worked example, with explicit assumptions. Suppose a production feature serves an LLM self-hosted on a single cloud GPU instance. Assumed instance cost: roughly $3 per GPU-hour — illustrative, use your provider’s actual rate. Assumed sustained throughput: the deployment generates on the order of 2,000 output tokens per second at the target batch size and precision — this is an example figure; real throughput must be measured for your model and stack, not assumed. Cost per output token = instance cost per second ÷ tokens per second = ($3 / 3,600 s) ÷ 2,000 tokens/s ≈ $0.00000042 per output token, or roughly $0.42 per million output tokens under these assumptions. If a single request generates 500 output tokens, its serving cost is on the order of $0.0002 — trivial in isolation, and exactly the figure that becomes significant when multiplied across millions of requests a month. That multiplication is the whole point: a per-request cost that looks like a rounding error becomes a gross-margin line at scale. The figures above are illustrative arithmetic, not a benchmark. The throughput number in particular is the one teams most often assume rather than measure — and assuming it is how cost-per-request projections go wrong. Establishing the real number requires profiling the serving path itself, which connects this concept to GPU-level performance profiling and instrumentation rather than spreadsheet estimates. The discipline of measuring cost, efficiency, and value as distinct quantities — rather than reading a raw cloud bill — is treated directly in the distinction between cost, efficiency, and value in AI hardware. How Does Inference Connect to Cost-Per-Request for a Feature? Directly. Once you accept that each request consumes a measurable quantity of tokens or GPU-seconds, cost-per-request is just that quantity multiplied by a unit rate, plus any fixed overhead allocated per request. That number is the bridge between inference behaviour and gross margin per AI feature. The chain is: tokens-or-GPU-seconds-per-request → cost-per-request → cost-per-active-user → gross margin on the feature. Break the chain at the first link and every downstream number is a guess. We see this regularly in early-stage AI products — the cost-per-request baseline was never established, so no one can say whether usage growth improves or destroys the unit economics. Building that baseline, and the broader practice of treating an AI feature as a unit-economics object, is the subject of unit economics for production AI in practice. Why Do Inference Costs Grow Over Time? Because inference cost is a function of usage, and usage of a successful feature grows. Training is done; the weights are fixed; that cost is behind you. But every new active user, every increase in average session length, every richer prompt adds inference load. A feature that was cheap to serve at launch can become the dominant compute line as it succeeds — the better it does, the more it costs. There are second-order growth drivers too: prompt templates tend to get longer as teams add context and few-shot examples, output lengths creep up as features get more conversational, and retrieval-augmented patterns inject more tokens per call. None of these touch the training budget. All of them raise cost-per-request, silently, unless someone is watching the meter. FAQ What is inference in AI? Inference is the phase where a trained model takes an input it has not seen and produces an output using its fixed parameters. No weights change during inference — the only work is the forward pass that turns an input into a prediction. It is what a model does every time a user actually uses an AI feature. How does inference differ from training in an AI workload? Training is a one-time (or periodic) capital-like spend that produces the model’s weights, involving both forward and backward passes plus gradient updates. Inference is a recurring operating expense that runs the forward pass only, every time the feature is used. A model that was cheap to train can be expensive to serve, and vice versa, which is why treating them as one “model cost” misleads. Why does every inference request carry a recurring cost? Because each genuinely novel request does real, billable work — the model must actually run a forward pass, occupying hardware for a measurable interval. Caching helps for identical inputs, but new prompts, images, or user contexts cannot be served for free. The meter resets to zero only when no one is using the feature. What consumes resources during an inference call — tokens, GPU-seconds, or API calls? It depends on the deployment. LLM serving consumes input and output tokens, with output tokens usually dominating because each requires its own forward pass. Self-hosted models consume GPU-seconds — the wall-clock time a request occupies an accelerator. Managed endpoints bill per call or per token, where the same per-request logic applies but the meter is the provider’s. Where does inference run in a deployed AI serving path? Inference runs at whichever placement the serving path chooses: a managed API endpoint, self-hosted on rented cloud GPUs, self-hosted on owned hardware, or on the user’s device at the edge. Each placement turns inference cost into a different financial object — a per-token charge, a GPU-second rate against a utilisation curve, or an amortised fixed cost. How is inference cost per token calculated for an LLM serving a production feature? For a self-hosted model, divide the instance cost per second by the sustained tokens generated per second. For example, a roughly $3/GPU-hour instance generating on the order of 2,000 tokens per second works out to about $0.42 per million output tokens — though the throughput figure must be measured for your specific model and stack, not assumed. Why do inference costs tend to grow over time as a feature scales? Because inference cost is a function of usage, and a successful feature’s usage grows while its training cost stays behind you. More active users, longer sessions, longer prompts, and retrieval-augmented patterns all raise cost-per-request without touching the training budget. The better a feature performs, the more it costs to serve — unless someone is tracking the meter. Where This Leaves You The reason this primer matters is not the definition itself — most engineers can define inference. It is what the definition forces you to admit: that you have a per-request cost event running in production whose unit cost you may never have measured. The conceptual grounding here exists so that the cost number has something to stand on. Establishing what inference is comes before profiling the serving path that decides what it actually costs, and well before any conversation about cutting it. If you take one operating question from this: for each AI feature you run, can you state its cost-per-request as a measured number rather than a guess? If the answer is no, the meter is still running — you simply are not reading it. That gap, and the disciplined work of closing it, is where the ai infrastructure for SaaS practice starts.
