A distilled model is often sold as “the same model, cheaper.” That framing survives a marketing slide and collapses in a procurement committee. Distillation does not copy a model wholesale; it transfers capability selectively, and the places where the transfer is incomplete are exactly the cases a buyer cares about most: their edge cases, their rare intents, their long-tail failure modes. Carry the teacher model’s leaderboard scores into the decision and you are defending a number that was never measured on the model you are actually buying. That is the core problem this article is about. A distilled candidate is not evaluated by inheriting its teacher’s evidence. It is re-evaluated on your task, your data, and your risk tolerance — or it should not enter the comparison at all. What matters most about distillation training in practice? Knowledge distillation trains a smaller “student” model to reproduce the behaviour of a larger “teacher” model. Instead of learning only from hard labels (the correct answer), the student learns from the teacher’s full output distribution — the soft probabilities the teacher assigns across all possible tokens or classes. Those soft targets carry more information than a one-hot label: they encode how confident the teacher was, and which alternatives it considered plausible. That richer signal is what lets a much smaller network approximate a much larger one on a surprising range of tasks. The result is a model with a fraction of the parameters, lower memory footprint, and lower inference cost-per-decision. This is why distillation has become a standard lever in production LLM stacks — you can serve a distilled student on cheaper hardware, at lower latency, in configurations where the teacher would be uneconomic. Frameworks like PyTorch and Hugging Face’s transformers make the mechanics accessible; the harder part has never been running distillation, it has been knowing what you gave up. Because that is the practical meaning of distillation: it is a lossy transfer, and the loss is not uniform. The student learns the teacher’s behaviour on the training distribution well. It learns the tails — the inputs the teacher itself saw rarely — much less reliably. For a general explainer of the mechanism and where it fits a regulated evidence pack, see our companion piece on distillation in ML and how it slots into a regulated AI evidence pack. Here the focus is narrower: what the transfer does to the evidence you are allowed to claim. What is the difference between a teacher model and a distilled student model in terms of measured capability? On common cases, a well-distilled student can match its teacher closely enough that the gap is invisible on aggregate benchmarks. That is the whole point of distillation, and it is also the trap. Aggregate accuracy is a weighted average dominated by common inputs. If 95% of a benchmark’s items are common cases and the student matches the teacher on all of them, the student scores within a percentage point or two of the teacher — while diverging sharply on the 5% that the average buries. The divergence is a capability-transfer artifact, not noise. A common pattern we see: a student holds up on well-represented intents and degrades on rare, safety-relevant, or adversarial inputs — precisely the region where a procurement committee’s risk tolerance actually lives (observed across LLM-evaluation engagements; not a published benchmark). The teacher’s leaderboard rank tells you nothing about where that degradation sits, because the leaderboard number was never your number — it was measured on a public prompt distribution that does not resemble your workload. The practical consequence is that “teacher capability minus a small tax” is the wrong mental model. The correct model is: the student inherits the teacher’s average competence and re-samples its own competence on the tails. You cannot predict the tail behaviour from the average. You have to measure it. Why can’t a distilled model inherit its teacher’s evaluation evidence? Three reasons, and they compound. First, the evidence class is wrong. A teacher’s benchmark score is a benchmark-class claim about the teacher — a named, reproducible measurement on a named model. Transfer it to the student and you have silently upgraded an inference (“the student should behave similarly”) into a measurement it never earned. A committee that later discovers the student was never tested on the buyer’s data is entitled to reject the whole pack. Second, the divergence point is the buyer’s own prompt distribution — which no public leaderboard captures. Distillation loss concentrates on the tails of the teacher’s training distribution, but the buyer cares about the tails of the buyer’s distribution, and those two tail regions rarely coincide. A student can look fine on MMLU or Chatbot Arena and still misfire on the buyer’s domain-specific edge cases. The only way to know is to run the buyer’s evaluation set against the student directly. Third, distillation is a training intervention, so anything the teacher’s evidence certified about calibration, refusal behaviour, or failure-mode distribution has to be re-established. Confidence scores in particular do not survive distillation intact — the student’s probability outputs are its own, and how you read those confidence scores must be re-derived from the student, not assumed from the teacher. This is the difference between defending “it’s a distilled version of a strong model” at committee and defending “here is the distilled model’s measured accuracy and failure catalogue on our workload.” Only the second is decision-grade. Where does a distilled model typically diverge from its teacher on the buyer’s prompt distribution? The divergence is not random — it clusters in recognisable places. Use this as a diagnostic checklist when scoping the evaluation set for a distilled candidate. Distilled-candidate divergence checklist Rare intents. Inputs the teacher saw infrequently during its own training are the first to degrade under distillation. If your workload has a long tail of low-frequency but high-stakes intents, over-sample them in the eval set. Domain-specific vocabulary. Terminology, entity names, or formats specific to your domain (clinical, legal, financial) are common divergence points, because they were under-represented in the general distillation corpus. Adversarial and safety-relevant inputs. Refusal behaviour and jailbreak resistance are learned behaviours that distillation can dilute. Re-test them explicitly. Long-context and multi-step reasoning. Students frequently lose ground on chained reasoning before they lose ground on single-shot answers — the failure shows up on multi-step items, not on the aggregate score. What reasoning evals actually test is the relevant reference here. Calibration under uncertainty. Where the teacher expressed useful uncertainty, the student may express misplaced confidence — the most dangerous divergence, because it degrades silently. The through-line: the student’s weak spots are the buyer’s tail, and the buyer’s tail is invisible on any public leaderboard. This is why task-specific classification accuracy can diverge from leaderboard rank even when two models look nearly identical on aggregate. How should the procurement evidence pack represent a distilled candidate’s cost-versus-accuracy trade-off? This is where distillation earns its place in a comparison — but only if the pack states the trade-off in like-for-like terms. Distilled models often cut inference cost-per-decision by a large margin, which directly changes the cost line in a procurement comparison. The mistake is presenting the cost saving as settled while the accuracy delta stays hand-waved. The pack should carry a single row per candidate that pins cost and measured accuracy together, both on the buyer’s own load: Distilled-vs-teacher procurement comparison Dimension Teacher model Distilled student Evidence class Cost-per-decision Baseline Typically a large reduction — state the measured figure from your deployment, not the vendor’s benchmark — from the deployed candidate on your load Aggregate task accuracy Measured on buyer set Measured on buyer set (often within a point or two) benchmark — re-run, not inherited Tail / edge-case accuracy Measured on buyer edge set Re-measured on buyer edge set (this is where deltas surface) benchmark — re-run, not inherited Failure-mode catalogue Documented Re-documented for the student observed-pattern from evaluation runs Calibration / confidence behaviour Certified Re-certified for the student benchmark — re-run, not inherited Every “distilled student” cell that matters is re-measured, never inherited. Re-running task-specific evaluation on a distilled candidate typically converts an ambiguous “cheaper but risky” option into a like-for-like committee comparison in a single evaluation round — avoiding a deferred decision or a post-deployment rollback (observed across procurement-evaluation engagements; not a benchmarked rate). That single round is cheap relative to what a rollback costs. For the methodology of measuring student-versus-teacher deltas on your workload as an infrastructure decision, the vertical lens on AI infrastructure and SaaS carries the operational detail. What failure modes should be re-checked at the buyer’s risk tolerance? Risk tolerance is not a global setting; it is per-workload. A distilled model that is acceptable for internal drafting may be unacceptable for a decision that touches a regulated outcome. Re-check the failure modes that your specific risk tolerance makes material: silent miscalibration (confident wrong answers), degraded refusal on safety-relevant prompts, and reasoning collapse on multi-step tasks. Each of these can pass the aggregate score and fail the committee’s actual bar. The benchmark methodology for measuring teacher-versus-student deltas rigorously — the fairness and reproducibility side of the comparison — lives on the LynxBenchAI benchmarking methodology reference; that is where the measurement discipline sits, while the framing of what belongs in your evidence pack sits inside our AI governance and trust practice. FAQ How does distillation training work? Distillation trains a smaller student model to reproduce a larger teacher model’s full output distribution — its soft probabilities, not just the correct answers. The result is a smaller, cheaper model that approximates the teacher on common cases. In practice it is a lossy transfer: the student learns the common distribution well and the tails much less reliably. What is the difference between a teacher model and a distilled student model in terms of measured capability? On common cases a well-distilled student can match its teacher closely enough that aggregate benchmarks hide the gap. The difference concentrates on the tails — rare intents, adversarial inputs, multi-step reasoning — which the average buries. The student inherits the teacher’s average competence but re-samples its own competence on those tails, so you cannot predict tail behaviour from the aggregate score. Why can’t a distilled model inherit its teacher’s evaluation evidence for a procurement decision? Because the evidence class is wrong: a teacher benchmark is a measurement of the teacher, and transferring it silently upgrades an inference into a measurement the student never earned. Distillation loss concentrates where public leaderboards can’t see, and calibration and failure behaviour are training-dependent and must be re-established. Only the student’s measured accuracy and failure catalogue on the buyer’s workload is decision-grade. Where does a distilled model typically diverge from its teacher on the buyer’s own prompt distribution? Divergence clusters in recognisable places: rare intents, domain-specific vocabulary, adversarial and safety-relevant inputs, long-context and multi-step reasoning, and calibration under uncertainty. These are the buyer’s tail, and the buyer’s tail is invisible on any public leaderboard. Scope the evaluation set to over-sample exactly these regions. How should the procurement evidence pack represent a distilled candidate’s cost-per-decision versus accuracy trade-off? Pin cost and measured accuracy together in one row per candidate, both measured on the buyer’s own load. The cost-per-decision saving is often large, but present it beside re-measured aggregate accuracy, tail accuracy, failure-mode catalogue, and re-certified calibration — never inherited from the teacher. That single re-run converts a “cheaper but risky” option into a like-for-like comparison. What failure modes should be re-checked when evaluating a distilled model at the buyer’s risk tolerance? Re-check silent miscalibration (confident wrong answers), degraded refusal on safety-relevant prompts, and reasoning collapse on multi-step tasks. Risk tolerance is per-workload, so what is acceptable for internal drafting may fail the bar for a regulated decision. Each of these failure modes can pass the aggregate score and still fail the committee’s actual threshold. The question to carry into committee The useful question is not “is this a distilled version of a strong model?” — it is “what did the distillation cost us on the inputs we actually care about, and can we show it?” A distilled candidate that arrives with its own measured accuracy and its own failure catalogue on your workload is a real option you can defend. One that arrives wearing the teacher’s leaderboard scores is a deferred decision waiting to become a rollback. The failure class here is inherited-evidence: the fix is a re-run, scoped to your tail, folded into the evidence pack before the model reaches the table.
