Ask most teams for their model metrics and you get a familiar checklist: accuracy, precision, recall, F1, maybe a latency number bolted on at the end. Every one of those is real. None of them, on its own, tells you which deployment of a qualified model to ship. That gap is where money leaks. A serving config can clear every accuracy bar you set, pass the latency gate, and still lose money on every request it answers — because not one of the quality metrics is measured in the units your business actually bills in. The checklist qualifies a model for a task. It does not decide the config. Those are two different jobs, and conflating them is the most common way we see a production AI feature ship into a margin problem nobody predicted from the scorecard. Two metric families that keep getting merged into one The cleaner way to think about model metrics is to split them into two layers that answer two different questions. The first layer is model-quality metrics: accuracy, precision, recall, F1, and their task-specific cousins (BLEU for translation, nDCG for retrieval, exact-match for extractive QA). These qualify a model. They answer: is this model good enough at the task to be allowed anywhere near production? If a fraud classifier misses half the fraud, no serving config saves it. Quality metrics are a gate, and a model either clears the gate or it does not. The second layer is serving-economics metrics: p95 and p99 latency, throughput under realistic concurrency, cost-per-request, and — for token-generating models — cost-per-token. These decide the config. Once a model has cleared the quality gate, you are no longer choosing whether to ship it; you are choosing how. Which quantization? Which batch size? Which runtime — vLLM, TensorRT-LLM, SGLang? Which GPU, and how many replicas behind the autoscaler? Every one of those choices moves the economics metrics and leaves the quality metrics untouched (or nearly so). That is the whole point of the second layer: it varies the numbers the first layer holds fixed. The failure is treating one flat list as the scorecard. When accuracy and latency sit in the same table with no line between them, a reviewer reads down the column, sees green, and approves — without ever asking what the green config costs per request. The distinction between qualifying a model and configuring its deployment is the same one at the heart of spec-ing the compute behind a production AI feature, where the cost-per-request reality tends to arrive long after the accuracy sign-off. How does machine learning model metrics work in practice? In practice you read metrics in that order — quality first as a gate, economics second as a decision — and you never let the second borrow authority from the first. Concretely: a candidate model is evaluated on a held-out set for its quality metrics. Suppose it hits the accuracy and F1 thresholds the task owner set. Good. That result is now fixed. It does not improve or degrade meaningfully across most serving configs — a well-behaved INT8 quantization of a qualified model typically holds its task accuracy within a small margin (an observed pattern across the optimization work we do; the exact drift is model- and calibration-specific, not a benchmarked constant). So you stop optimizing the quality number and start measuring the economics ones against each candidate config. This is where a lot of scorecards quietly go wrong. Teams keep reporting accuracy as if it were still the live variable, when the live variable has become cost-per-request. Accuracy is settled. The decision in front of you is which of three qualified configs to run, and accuracy cannot rank them because it is (approximately) the same across all three. The metric that discriminates is the one measured in the unit you bill in. Why a config that passes every accuracy metric can still lose money Here is the mechanism, because it is not obvious from the checklist. Cost-per-request is a function of three things the quality metrics never touch: how much compute a single request consumes, how efficiently that compute is packed (batching, concurrency, GPU utilization), and what that compute costs per hour. A model can be accurate and expensive — a large reasoning model that emits 4,000 tokens of chain-of-thought to answer a question that a smaller model answers in 200 tokens has the same or better accuracy and several times the cost-per-request. The accuracy column says “ship it.” The cost-per-token column says “not at this price.” For token-generating models the trap is sharper still, because cost scales with output length and output length is not on the quality scorecard at all. A model that “passes” on answer correctness can bankrupt a feature on verbosity. Estimating this before you commit is exactly what an LLM token calculator does for inference cost-per-request — it turns the invisible token count into the billable unit. The other half of the mechanism is utilization. Two configs of the same qualified model, same GPU, same accuracy — one at batch size 1 and one at batch size 32 — can differ by an order of magnitude in requests served per GPU-hour, and therefore in cost-per-request. Nothing in the quality metrics registers that. You only see it when you measure p95 latency and throughput together at realistic concurrency, which is why the per-config latency and utilization measurements from GPU profiling are what turn a serving-economics metric from an abstraction into a real cost number. Compiler and runtime choices move the same lever — machine learning compilers cut cost-per-request in production precisely by changing how much billable compute a fixed-quality model consumes. A two-layer metric scorecard The fix is structural: stop presenting one flat table and present two layers, with the gate above the decision. The point of the layout is that a reviewer physically cannot approve a config on the strength of the top layer alone, because the top layer is identical across the candidates. Quality gate (qualifies the model — same across configs) Metric Class Role Accuracy / F1 / task metric quality Pass/fail gate. Model qualifies or it does not. Precision / recall balance quality Confirms the error profile matches the task cost of false positives vs negatives. Task-specific score (nDCG, BLEU, exact-match) quality Confirms fitness for the specific job, not a generic leaderboard rank. Once every candidate config clears this layer, it stops discriminating. Every row below is now the live decision. Serving-economics decision (decides the config) Metric Class Config A (INT8, bs=8) Config B (FP16, bs=32) p95 latency operational measurement within SLO within SLO Throughput (req/GPU-hr) operational measurement lower higher Cost-per-request operational measurement higher lower Cost-per-token (gen models) operational measurement higher lower The illustrative numbers above are directional placeholders — the shape is what matters. Both configs pass the gate and both meet the latency SLO, so accuracy and latency alone would call it a tie. The tiebreaker is cost-per-request, and it is only visible because the two layers are separated. That separation is what keeps a benchmark honest: it prevents a quality pass from being read as a deployment recommendation. If you want to see this scorecard applied to a live deployment path, the [inference cost-cut sprint](Inference Cost-Cut Pack) measures the serving-economics layer against your current config to produce a real before/after cost-per-request — not a tokens-per-second headline. It is the same lens we bring to platform teams building on AI-infrastructure SaaS, where margin per request is the number the board actually asks about. Which metrics qualify, and which decide? A short diagnostic you can run against your own scorecard today: Does your approval table draw a line between quality metrics and economics metrics? If accuracy and cost-per-request sit in the same undifferentiated list, the line has been erased and configs are probably being chosen on the wrong layer. Is cost-per-request (or cost-per-token) on the scorecard at all? If the deployment decision is being made on throughput or accuracy headlines, the billable unit is missing and the margin risk is unmeasured. Are you still optimizing the quality number after the model has cleared the gate? If so, effort is going into a settled variable while the live variable — the config economics — goes unmeasured. Answering “no” to the first two is the exact gap we watch for: teams that pass every accuracy bar and still cannot say what a request costs. FAQ How does machine learning model metrics work? Model metrics work in two layers read in order: quality metrics act as a gate that qualifies a model for a task, and serving-economics metrics act as the decision that picks which deployment of a qualified model to ship. In practice, once a model clears the quality gate its accuracy is fixed, and the live variable becomes cost-per-request. Reading metrics in that order — and never letting the economics layer borrow authority from the quality layer — is what keeps the scorecard honest. What is the difference between model-quality metrics and serving-economics metrics? Model-quality metrics (accuracy, precision, recall, F1) measure how good a model is at the task and are approximately constant across serving configs. Serving-economics metrics (p95 latency, cost-per-request, cost-per-token) measure what a deployment costs and vary strongly with quantization, batch size, runtime, and hardware. The first family qualifies the model; the second decides the config. Which metrics qualify a model for a task, and which decide the serving configuration you ship? Quality metrics qualify: accuracy, F1, precision/recall balance, and task-specific scores like nDCG or BLEU form a pass/fail gate. Economics metrics decide: p95 latency, throughput per GPU-hour, cost-per-request, and cost-per-token rank the qualified configs. Accuracy cannot rank configs because it is roughly identical across them. Why can a config that passes every accuracy metric still lose money per request? Because cost-per-request depends on compute consumed, packing efficiency, and hardware price — none of which appear on the quality scorecard. A model can be accurate and verbose, emitting many more tokens than a cheaper alternative to reach the same correctness, so it passes on accuracy and fails on cost-per-token. Utilization differences between batch sizes can also swing cost-per-request by an order of magnitude with no change in accuracy. How do you assemble a metric scorecard that separates model quality from deployment margin? Present two layers, not one flat list: a quality gate on top (accuracy, F1, task score) and a serving-economics decision below (p95 latency, throughput, cost-per-request, cost-per-token). Every candidate config that clears the gate becomes a row in the lower table, where cost-per-request is the tiebreaker. The physical separation stops a reviewer approving a config on quality alone. How do cost-per-request and cost-per-token relate to standard model-quality metrics, and where do accuracy-based metrics stop short? Cost-per-request and cost-per-token are measured in the units the business bills in; accuracy-based metrics are not. Accuracy tells you whether a model is fit for the task but says nothing about how much compute a request consumes or how efficiently it is packed. Accuracy-based metrics stop short exactly at the deployment decision, because they are effectively constant across the configs you are choosing between. What does a two-layer metrics table look like for comparing two serving configurations of the same qualified model? The top layer lists quality metrics that read identically for both configs — confirming both are qualified. The bottom layer lists p95 latency, throughput, cost-per-request, and cost-per-token, where the two configs diverge. When both meet the latency SLO, cost-per-request is the deciding row; the layout makes that decision visible instead of hiding it inside a single mixed column. Once your scorecard is split this way, the open question stops being “is the model good enough” and becomes “which qualified config defends the margin” — and that question is answered in cost-per-request, measured against your deployed path, not in an accuracy headline read off a leaderboard.
