“It’s a 1-bit model, so it costs a fraction to run” is the sentence that starts most of these conversations. It is also the sentence that gets a procurement committee into trouble, because the phrase compresses two very different claims into one word: that the model is cheaper to serve (usually true) and that it will do your task about as well as the full-precision version you were going to buy (unknown, and often false). Extreme quantisation is a real, useful technique. The mistake is assuming the leaderboard delta between a 1-bit model and its full-precision sibling transfers to your decision. It almost never transfers cleanly, and the gap is exactly in the dimensions a committee cares about. Let’s be precise about what “1-bit” even means, because the name is doing some rounding of its own. How does a 1-bit LLM actually work? A conventional LLM stores each weight as a 16-bit or 32-bit floating-point number. Quantisation reduces that precision — 8-bit and 4-bit quantisation are now routine, and libraries like bitsandbytes and formats like GGUF make them easy to apply. “1-bit” pushes this to the extreme. In the architecture popularised by Microsoft Research’s BitNet work, weights are constrained to the set {-1, 0, +1}. That is three possible values, which needs about 1.58 bits to represent — hence the “1.58-bit” label you’ll see alongside “1-bit”. The marketing rounds down; the information theory does not. The point of doing this is that once weights are ternary, the expensive floating-point multiply-accumulate operations that dominate transformer inference can be replaced by additions and sign flips. That changes the arithmetic profile of the model, not just its storage footprint. Memory bandwidth — usually the binding constraint on inference throughput — drops sharply because each weight occupies a fraction of the space. On the right hardware and with a kernel written for ternary matmul, the throughput gains are substantial. So the cost story is genuinely attractive. The problem is that the same operation that makes the model cheap also changes what the model is. You have not compressed a model; you have trained (or retrained) a different model that happens to share an architecture family with the original. Its behaviour on your inputs is an open question, and a public benchmark is a poor place to look for the answer. Why can’t a 1-bit model’s public leaderboard score answer whether it fits my workload? A leaderboard reports one aggregate number over a fixed, public prompt distribution. When a 1-bit model posts a score “within a few points” of a full-precision baseline on a general suite, that comparison is true for the suite and silent about everything else. Quantisation error is not uniform across tasks. It concentrates in the places where the model was relying on fine-grained weight distinctions — long-context reasoning chains, low-frequency vocabulary, structured-output formatting, numerically sensitive tasks. If your workload lives in one of those places, the leaderboard’s aggregate hides the regression you will actually feel. This is the same divergence we describe in why the leaderboard number isn’t your number: a general benchmark measures general performance, and your procurement decision is not a general question. The 1-bit case makes it sharper because the compression specifically redistributes error, so the leaderboard-to-workload gap is not just noise — it is systematically larger for exactly the demanding tasks that justified buying an LLM in the first place. When a buyer walks a 1-bit leaderboard score into a committee, the predictable question comes back: what about our prompt distribution, our load, our failure tolerance? The general benchmark cannot answer any of the three. That is not a flaw in the benchmark — it was never built to answer them. It is a flaw in treating the benchmark as procurement evidence. We treat a 1-bit candidate exactly the way we treat any other model in a procurement-grade eval, through the AI governance and trust discipline: evaluate it on the buyer’s own workload, because compression changes precisely the things a committee is being asked to approve. What changes when a model is quantised to ~1–1.58 bits? Three profiles shift together, and they don’t shift in lockstep. Understanding which one moves most for your task is the whole evaluation. Dimension Typical direction under ~1.58-bit quantisation Why a committee cares Task accuracy Down, unevenly — small on easy tasks, larger on reasoning, formatting, and rare-token tasks Determines whether the cheaper model still meets the acceptance bar Latency / throughput Better — lower memory footprint and cheaper arithmetic raise throughput The reason the option is on the table at all Failure modes Different, not just more frequent — new error shapes (silent formatting drift, degraded refusals, confident-but-wrong on edge cases) Failure shape drives operational risk and downstream review cost The failure-mode row is the one buyers underestimate. It is intuitive to think a compressed model simply makes the same mistakes more often. That is not what we see. Quantisation tends to change the character of the errors — a model that reliably produced valid JSON at full precision may start emitting subtly malformed structure under load, or a safety refusal that was crisp becomes hedged. An aggregate accuracy score cannot surface this, because the score treats all errors as equivalent. Your review pipeline does not; a new error shape can cost more to handle than a slightly higher error rate of a familiar kind. These claims are observed-pattern: the direction of each shift is consistent across the quantisation work we’ve evaluated, but the magnitude is entirely task-dependent and is not a benchmarked rate you should port to your own numbers. That task-dependence is the reason a general score is unsafe to reuse. How should a 1-bit candidate be evaluated inside a procurement-grade evidence pack? The discipline is not special to 1-bit models. It is the same procurement-evidence lens we apply to any candidate: you do not exempt a model from the evidence pack because it is cheaper to run. A 1-bit candidate is a candidate, and it earns approval by producing the same three artifacts against the same baseline. Diagnostic checklist — evaluating a 1-bit candidate Run it on your prompt distribution, not a public suite. Use a held-out sample of your real traffic, labelled to your acceptance criteria. Report task accuracy against the full-precision baseline on that set. Measure cost-per-decision under your actual load. Throughput gains only materialise at your batch sizes, sequence lengths, and concurrency. Benchmark the deployed configuration, not the vendor’s. Characterise the failure-mode delta, not just the accuracy delta. Categorise the errors the 1-bit model makes that the baseline does not, and estimate the downstream review cost of each new shape. Confirm the kernel and hardware path. Ternary matmul speedups depend on a kernel that actually exploits them; without it you keep the accuracy hit and lose the cost win. Verify the serving stack (the quantised runtime, the GPU) delivers the arithmetic you paid for. Package all three as a like-for-like comparison. Accuracy, cost-per-decision, and failure delta, side by side with the standard model, in one document a committee can decide on in a single round. Steps 2 and 4 are where the numbers get measured against real infrastructure — the vertical procurement-eval methodology that pairs with this evidence discipline. The point of the pack is that it lets a committee approve or reject in one round instead of discovering the accuracy regression after deployment, which is the expensive failure this whole exercise exists to prevent. How is cost-per-decision represented so a committee can compare like-for-like? Cost-per-decision is the unit that makes a 1-bit option and a full-precision option commensurable. Raw throughput is not — a model that is twice as fast but wrong twice as often on your task has not saved you anything once you count the cost of handling the extra errors. Cost-per-decision folds the inference cost and the downstream consequence of the accuracy and failure-mode deltas into a single figure the committee can weigh against the standard model. Concretely, for each candidate you report the compute cost to produce one accepted decision at your load, plus the expected review or remediation cost attributable to its error profile. For a worked frame: if a 1-bit model runs at, say, roughly three times the throughput of the baseline but introduces a new malformed-output failure mode that requires manual review on a small fraction of outputs, the like-for-like question is whether the compute saving outweighs the added review cost at your volume. That arithmetic is specific to your load and your review economics — which is exactly why the number has to come from your workload, not a leaderboard. The same reasoning underlies how we read distillation candidates for procurement: a smaller or cheaper model is only a saving if the total cost per accepted decision goes down. When is a 1-bit model a genuinely good procurement choice? Often enough that dismissing the technology would be a mistake. A 1-bit model tends to be the right call when your task is tolerant of the error shift — high-volume classification, routing, or extraction where the accuracy bar is comfortably above the regression, and where the throughput gain directly reduces a real infrastructure bill. If cost is the binding constraint and your acceptance criteria have headroom, the cheaper inference is worth it, and the evidence pack will show it. It is the wrong call when your workload sits in the tasks where quantisation error concentrates — multi-step reasoning, strict structured output, safety-critical refusals, rare-domain vocabulary — and your failure tolerance is low. There the accuracy regression and the new failure shapes eat the savings, sometimes several times over. The technology is real and can be worth it; the decision is never “1-bit is cheaper” in the abstract. It is “1-bit is cheaper for this task at this load with this failure tolerance,” and only your own evaluation closes that sentence. FAQ How should you think about 1-bit llm in practice? A 1-bit LLM constrains its weights to a very small set — in the common BitNet-style approach, the ternary set {-1, 0, +1}, which needs about 1.58 bits, so “1-bit” is a rounding of “1.58-bit”. This lets expensive floating-point multiplies be replaced by additions and sign flips and cuts the memory footprint sharply, raising inference throughput. In practice you are serving a different model, not a compressed copy of the original, so its behaviour on your inputs must be verified rather than assumed. Why can’t a 1-bit model’s public leaderboard score answer whether it fits my workload? A leaderboard reports one aggregate number over a fixed public prompt distribution, while quantisation error is not uniform — it concentrates in demanding tasks like reasoning, structured output, and rare vocabulary. So a score “within a few points” of the full-precision baseline can hide a large regression on exactly your task. The benchmark was never built to answer questions about your prompt distribution, your load, or your failure tolerance. What changes in accuracy, latency and failure modes when a model is quantised to ~1–1.58 bits? Task accuracy drops unevenly — small on easy tasks, larger on reasoning and formatting; latency and throughput improve because of the lower footprint and cheaper arithmetic; and failure modes change shape rather than just frequency. The new error shapes — silent formatting drift, hedged refusals, confident-but-wrong edge cases — are the part buyers most underestimate because an aggregate accuracy score treats all errors as equivalent while your review pipeline does not. How should a 1-bit candidate be evaluated inside a procurement-grade evidence pack? Treat it like any other candidate: run it on a held-out sample of your real, labelled traffic; measure cost-per-decision at your actual load; characterise the failure-mode delta versus the full-precision baseline; confirm the serving kernel and hardware actually deliver the ternary speedup; and package all three as a like-for-like comparison. The point is to let a committee approve or reject in one round instead of discovering a regression after deployment. How is cost-per-decision for a 1-bit model represented so a committee can compare it like-for-like against a full-precision model? Report the compute cost to produce one accepted decision at your load, plus the expected review or remediation cost attributable to the model’s error profile, side by side with the standard model. Raw throughput is not commensurable — a faster model that is wrong more often on your task has not saved anything once you count error handling. Because both terms depend on your load and review economics, the figure must come from your workload, not a leaderboard. When is a 1-bit model a genuinely good procurement choice, and when is the cheaper inference not worth the accuracy trade? It is a good choice when the task tolerates the error shift — high-volume classification, routing, or extraction where the accuracy bar sits comfortably above the regression and the throughput gain cuts a real bill. It is the wrong choice when the workload lives in reasoning, strict structured output, or safety-critical refusals with low failure tolerance, where the regression and new failure shapes eat the savings. The decision is always task-, load-, and tolerance-specific, which only your own evaluation can settle. The honest closing question for any 1-bit candidate is not “is it cheaper to run” — it almost always is — but “does the cost per accepted decision on our workload actually fall once the accuracy regression and the new failure shapes are priced in?” That is a workload-selection question, and it is the one a leaderboard cannot answer for you.
