A procurement committee opens the shortlist deck and there it is: Llama 2 70B, with its MMLU score printed in bold next to a green checkmark. The implication is that the number settles the question — the model is “capable enough,” so it can move to the next round. It does not settle anything. A public capability score tells you the model can reason across a broad academic distribution. It says almost nothing about whether it will hold up on the buyer’s actual task, at the buyer’s latency budget, under the buyer’s risk tolerance. That gap — between a general capability signal and a task-specific verdict — is where most open-weight model evaluations quietly go wrong. Llama 2 70B is a useful case to reason through because it is one of the most-cited open-weight models in shortlist decks. It is a 70-billion-parameter model you can self-host, so its published scores show up everywhere buyers compare “open” against “hosted.” Understanding exactly what those scores prove, and where they stop, is the difference between an evidence pack a committee can act on and one that gets the decision deferred. How Llama 2 70B works, and what that means in practice Llama 2 70B is a decoder-only transformer with roughly 70 billion parameters, released with open weights that let you download, run, and fine-tune it on your own infrastructure. The practical consequence of that architecture is not the parameter count itself but what it enables: you can inspect the model, pin a specific version, run it inside your own trust boundary, and avoid sending prompts to a third-party API. For a regulated buyer, that self-hosting option is often the entire reason the model is on the shortlist. Running it is not free of engineering weight. A 70B model in half precision needs on the order of 140 GB of weights in memory before you account for the KV cache, so a single-GPU deployment usually implies quantisation or a multi-GPU setup with tensor parallelism across NVLink. Serving frameworks such as vLLM or TensorRT-LLM handle the batching and paged attention that make throughput viable under real load. None of this is exotic — it is the standard cost of hosting a large open-weight model — but it is exactly the part a headline benchmark number hides. The score tells you the model can reason; it does not tell you what it costs to serve at your concurrency. What Llama 2 70B’s public benchmark scores actually measure When a shortlist deck cites Llama 2 70B, it is almost always quoting a small set of public suites: MMLU for broad knowledge, ARC and HellaSwag for commonsense reasoning, maybe a coding or math suite. These are real, reproducible measurements — benchmark-class evidence, published by Meta and reproduced by third parties on fixed, public test sets. That reproducibility is genuinely valuable. It is why they exist. What they measure is general capability against a standardised distribution. MMLU asks multiple-choice questions across 57 academic subjects. A strong score means the model has broad latent knowledge and can follow the format. It does not mean the model will classify your support tickets correctly, extract the right fields from your contracts, or refuse the prompts your compliance team cares about. The distribution the benchmark samples from is not your distribution — and the gap between the two is unmeasured by the leaderboard, which is precisely the point developed in our walkthrough of why the leaderboard number isn’t your number. Three things a public score genuinely proves, stated as extractable claims: Llama 2 70B has broad academic knowledge sufficient to score competitively on MMLU-style multiple-choice suites (benchmark, per Meta’s published Llama 2 evaluation). Its reasoning ceiling as an open-weight model sits in a known band relative to other open models of its generation, which is useful context for what an open, self-hostable model can do (benchmark). Because the weights are public and the test sets are fixed, any independent party can reproduce the score — a property hosted-API models often cannot offer (observed-pattern across our evaluation engagements). What no public score proves: accuracy on your prompt distribution, behaviour under your prompt templates, failure modes at your risk tolerance, or cost-per-decision at your load. Those are properties of the deployment, not the model card. Where Llama 2 70B belongs in a procurement evidence pack The mistake is not citing Llama 2 70B. The mistake is citing it as the verdict. In a well-formed evidence pack, its public scores appear as one calibrated capability-context row — a signal that says “this open-weight model clears a general reasoning bar” — tied explicitly to the buyer’s task and deployment requirements. It sits alongside, never in place of, the metrics the committee actually acts on. The following table separates the two, because conflating them is the single most common reason an evidence pack fails review. Decision surface: what proves capability vs. what proves fit Evidence element What it establishes Evidence class Belongs as Llama 2 70B MMLU / ARC score General reasoning capability of an open-weight model benchmark (public) Context row — capability ceiling Reproducibility of the public score Open weights allow independent verification observed-pattern Context row — trust property Task accuracy on buyer’s prompt set Whether the model does your job benchmark (your test set) Decision metric — primary Failure-mode catalogue at your risk tolerance What the model does when it is wrong observed-pattern Decision metric — primary Self-hosting cost-per-decision under real load Whether the deployment is affordable observed-pattern / benchmark Decision metric — primary The rows in the top half provide context. The rows in the bottom half decide the procurement. A committee that arrives with only the top half — a headline number and a green checkmark — has brought background reading, not evidence. In our experience across LLM evaluation engagements, that is the pack that gets the decision deferred, adding a full approval round while the team scrambles to produce task-specific numbers they should have led with. The same failure pattern shows up whether the shortlisted model is open or hosted; we treat it structurally in our survey of what benchmark suites prove and where they fall short for procurement. How should a committee weigh public rankings against task-specific accuracy? Weight them by what they can answer. A public ranking answers “is this model in the right capability class at all?” — a useful gate that keeps you from shortlisting something structurally too weak. Task-specific accuracy on your own prompt distribution answers “will this model do the job we are buying it for?” — the question the purchase actually turns on. When the two disagree, the task-specific number wins, every time, because it measures the thing you are paying for. The practical rule we apply: a public score can disqualify a model but should never, on its own, qualify one. If Llama 2 70B scored far below your task’s minimum plausible capability, that is a legitimate reason to drop it early. But a strong public score is permission to run your own evaluation, not a substitute for it. This is the same asymmetry that governs how public leaderboard rank relates to real task fit — we develop it further in our discussion of what public LLM leaderboards actually measure, and it is why an evidence pack that leans on Elo or MMLU alone reads as incomplete to a careful reviewer. Building the task-specific number is the work. It means assembling a held-out set drawn from your real prompt distribution, running Llama 2 70B against it under the prompt templates and system messages you will actually deploy, scoring the outputs with metrics your domain cares about, and cataloguing the failures. That evidence is what governance reviewers need, and it is the throughline of our broader approach to AI governance and trust — the discipline of turning model behaviour into decision-grade documentation. What deployment and cost factors enter the evidence pack? Self-hosting Llama 2 70B is an infrastructure decision as much as a model decision, and the evidence pack has to reflect that. The relevant figures are not on the model card. They come from measuring the model on your target hardware: tokens-per-second at your concurrency, GPU-hours per thousand decisions, memory headroom after the KV cache, and the tail latency your users will actually experience under load. A model that scores well but costs three times your budget per decision is not a viable choice, and the committee needs to see that before it approves. These numbers connect the capability signal to real economics, which is why the procurement-eval methodology in our AI-infrastructure vertical treats deployment cost as first-class evidence rather than an afterthought. A quantised Llama 2 70B on a single high-memory GPU behaves very differently — in both quality and cost — from a full-precision multi-GPU serve, and the evidence pack has to state which configuration the numbers describe. Reporting an accuracy figure without naming the precision and serving stack it was measured under is the deployment equivalent of citing a leaderboard rank: technically true, operationally meaningless. Where does citing Llama 2 as context stop and task-specific methodology begin? The line is clean once you name it. Citing Llama 2 70B’s public scores is context: it establishes that an open-weight model can reach a certain reasoning bar and can be independently verified because the weights are public. Defining task-specific benchmark methodology is the decision work: it establishes whether this model, at your precision and load, on your data, at your risk tolerance, does the job. The first is one row in the pack. The second is the pack. FAQ How does Llama 2 70B work in practice? Llama 2 70B is a decoder-only transformer with roughly 70 billion parameters, released with open weights you can download, self-host, and fine-tune inside your own trust boundary. In practice that self-hosting option is often the reason it makes a shortlist, but serving it is not trivial — the weights alone need on the order of 140 GB in half precision, implying quantisation or a multi-GPU setup with a framework like vLLM or TensorRT-LLM. What do Llama 2 70B’s public benchmark scores actually measure, and what capability do they and don’t they capture? They measure general capability against standardised public test sets — MMLU for broad knowledge, ARC and HellaSwag for commonsense reasoning. A strong score proves broad latent knowledge and format-following, and because the weights and test sets are public, the score is independently reproducible. It does not capture accuracy on your prompt distribution, behaviour under your prompt templates, failure modes at your risk tolerance, or cost per decision at your load. Where does Llama 2 70B belong in a procurement-grade LLM evaluation evidence pack, and where does citing it mislead? It belongs as one calibrated capability-context row — a signal that an open-weight model clears a general reasoning bar — tied to the buyer’s task and deployment requirements. It misleads the moment it is presented as the verdict: a headline benchmark number with a green checkmark is background reading, not the task-specific accuracy, failure catalogue, and cost evidence a committee acts on. How should a committee weigh Llama 2 70B’s public rankings against task-specific accuracy on the buyer’s own prompt distribution? Weight each by what it can answer. A public ranking can legitimately disqualify a model that is structurally too weak, but it should never qualify one on its own; a strong score is permission to run your own evaluation, not a substitute for it. When public rank and task-specific accuracy disagree, the task-specific number wins because it measures the job you are actually buying. What deployment and cost factors matter when self-hosting Llama 2 70B, and how do they enter the evidence pack? The relevant figures are measured on your target hardware, not read off the model card: tokens-per-second at your concurrency, GPU-hours per thousand decisions, memory headroom after the KV cache, and tail latency under load. They enter the pack as first-class evidence, and any accuracy figure must name the precision and serving stack it was measured under, since a quantised single-GPU serve behaves very differently from a full-precision multi-GPU one. Where does using Llama 2 70B benchmarks as context stop and defining task-specific benchmark methodology begin? Citing the public scores is context — it establishes an open-weight capability ceiling and the trust property of reproducibility. Task-specific methodology is the decision work — assembling a held-out set from your real prompt distribution, running the model under your deployment templates, scoring with your domain’s metrics, and cataloguing failures at your risk tolerance. The first is one row in the pack; the second is the pack. When a shortlist deck leads with Llama 2 70B’s MMLU score, the right question is not “how high is it?” but “what does this model do on our data, at our latency budget, under our risk tolerance?” — the question a public capability leaderboard was never built to answer, and the one your evidence pack exists to close.
