Local LLM benchmarks are useful for building a shortlist, but they cannot make the decision for you. Published speed figures tell you how fast a card streams tokens from a given model. Public quality scores tell you how that model did on someone else’s questions. Neither tells you whether a fine-tuned 70B model on your hardware summarises your documents well enough, at a low enough cost per request, to take volume away from a hosted frontier model. That answer comes from two measurements you make yourself. The first runs your real query log through both systems, scores the outputs on your task metric and measures cost on the serving stack you would actually deploy. The second, if you quantise, is a parity test of the quantised checkpoint against its full-precision parent on the same held-out set.

The Bench360 authors reached the same conclusion from the systems side. They found no single best setup for local inference, and that the ideal choice demands a data-driven evaluation of model, quantisation and engine against specific deployment requirements.

Which local LLM benchmarks measure speed, and which measure quality?

Speed benchmarks describe how the serving system behaves. Quality benchmarks describe what it produces. A deployment decision needs both because quantisation trades accuracy for throughput.

Benchmark family	What it reports	Typical metrics
First-token latency	Time from submitting a prompt to receiving the first token of the response	TTFT, prompt-processing tokens/s
Generation speed	Time to produce each output token for one user, which is the speed that user perceives	TPOT, generation tokens/s
Serving under load	Total tokens per second across all concurrent users, up to the infrastructure’s limit	System TPS, TTFT and inter-token latency distributions
Distribution fidelity	How far a quantised model’s predictions drift from the FP16 reference	Perplexity, KL divergence
Task quality	Whether the outputs are correct for your task	Exact match, F1, ROUGE/BLEU, function-call accuracy

The trade-off shows up in measurements, not just in theory. In Bench360’s 24GB experiments, INT8 and INT4 quantisation let 2–4× larger models fit and often scored higher, but time per output token and energy rose by more than 350% for some configurations, notably Qwen2.5-32B INT4. An edge benchmarking study (ELIB, 2025) found that low-bit quantised models gave better throughput on every device tested, but those faster models showed poor accuracy. Even dedicated performance frameworks leave accuracy out of scope: the FMwork authors (2025) name accuracy as a key comparison aspect while excluding it from their own speed methodology.

Token generation at small batch sizes is limited by memory bandwidth, so achieved bandwidth predicts generation speed better than peak compute does. For interactive use, the FMwork paper uses 50 ms per token as a reference threshold for real-time interaction. Mozilla’s LocalScore treats a score of 1,000 as excellent, 250 as passable and anything below 100 as likely to give a poor experience. Those thresholds help you read a speed result. They say nothing about quality. For a fuller account of what a public score can and cannot capture, see our explanation of what an LLM benchmark measures.

Can public local LLM leaderboards pick the open-weight model for our documents?

No. A leaderboard can narrow your candidate list. It cannot choose a model for your documents, for three reasons.

The first is contamination. When benchmark test data leaks into a model’s training data, the reported scores overstate what the model can really do. A 2024 survey of the problem finds that automatic evaluation reduces the effect but cannot remove it, and human evaluation is also exposed. The second is retirement. Hugging Face retired the Open LLM Leaderboard in March 2025, after evaluating more than 13,000 models, because benchmarks have to change as model capabilities change. A score from a retired leaderboard is a historical record, not a measurement of the current field. The third is task mismatch. A leaderboard asks its own questions in its own format, while your analysts summarise your documents with your prompts, and nothing in a public ranking measures that distribution.

Leaderboards still earn a place upstream. They help pick which two or three open-weight families, quantisation formats and serving stacks are worth testing on your data. After the shortlist, the decision rests on your own data.

What hardware does a 70B model need, and how do we check a machine before committing?

A dense 70B model needs about 140GB just to hold its weights at 16-bit precision (70 billion parameters × 2 bytes). At Q4_K_M it needs roughly 40GB, and at Q3 about 30GB. Both are more than a single 24GB card holds. That arithmetic sets out the realistic routes.

Route	What the sources show	Where it fits
Single 24GB card, aggressive quantisation (Q2)	Indicative llama.cpp figures put an RTX 4090 at 18 tok/s and an RTX 3090 at 10 tok/s on 70B at Q2	Experiments only. Q2/IQ3 precisions are known to degrade reasoning on complex tasks
Single card with CPU offloading at Q4	Offloading over PCIe cuts throughput by more than 90% compared with running fully on the GPU	Batch work at most. Interactive use becomes painful
Two used RTX 3090s (48GB)	Holds a 70B model at Q4 without offloading, for less than a single used RTX 4090 costs	The cheapest consumer route to 70B at Q4. It adds capacity, not speed
Apple unified memory	The M3 Ultra runs 70B-class 4-bit weights at 13–49 tok/s with no offloading cliff	Offload-free local inference that favours model fidelity over peak speed
Datacentre GPUs	A dense 70B in FP8 needs two H100s in practice, because the weights fill one card and leave no KV-cache headroom. Llama-3.3-70B runs on two A100-80GB GPUs	Shared serving for many concurrent users

The dual-3090 route carries real build costs. It needs a 1200W or larger power supply, a motherboard with two usable x8 or x16 slots, serious airflow, and tensor-splitting configuration in llama.cpp or vLLM. Most used 3090s are out of warranty, so checking VRAM junction temperatures before trusting a card is part of the job.

We would profile the candidate machine before committing to a 70B deployment, not reason from rules of thumb. One inference-engineering guide makes the same recommendation: always measure end-to-end server performance. It reports a 2× latency difference it observed between nominally identical 8×A100 servers from two cloud providers. In practice, that means running llama-bench at your actual context depth and, for a serving box, a load sweep at your expected concurrency. Before you buy anything, LynxBenchAI’s published results show how each covered device runs the public PyTorch stack your own models also run on. That makes them a fair guide to how hardware compares, but they do not run or measure your workload. Card-against-card price-performance comparisons are a separate question from the one this page answers.

Which tools can we use to benchmark local LLMs on our own machine?

Each tool covers part of the picture. llama-bench, GuideLLM and LocalScore measure speed. Only a few touch quality.

Tool	What it measures	Notes
llama-bench (llama.cpp)	Prompt processing (pp), text generation (tg) and the two combined, as average tokens/s with standard deviation across repetitions	Excludes tokenisation and sampling time. Can prefill the KV cache to a set context depth
GuideLLM (vLLM project)	Full TTFT, inter-token latency and end-to-end distributions under synchronous, concurrent and rate-based load, with sweeps to find safe operating ranges	Built for a serving endpoint, not a single chat session
LocalScore (Mozilla)	Prompt-processing speed, generation speed and time to first token, combined into one score	A speed tool and public results repository, with no quality measurement
MLPerf Client (MLCommons)	TTFT and tokens/s on personal computers, with models quantised to int4	Each acceleration path must pass MMLU-based accuracy tests offline to qualify
llama.cpp perplexity tool	Perplexity and KL divergence of a quantised model against FP16 logits	The quality half of a quantisation check
Bench360	Task quality and system metrics (latency, throughput, energy, startup time) in one run, across several inference engines	Its published results cover models that fit in 24GB

Two cautions apply when comparing results across tools. One hardware comparison warns that figures from its two benchmark suites, which use different backends, are not interchangeable. And the backend can change quality itself: in the ELIB study, OpenCL GPU inference on two devices gave perplexity almost 10× worse than CPU inference on the same model.

Can a local 70B model match our hosted frontier model on our real workload?

It might, but no published source settles that for your documents, so the only reliable test is a side-by-side run on your own query log. The supplied evidence for open models beating a hosted frontier model is narrower than the question. The Laboratory-Scale AI study (FAccT 2024) found a fine-tuned open model can exceed GPT-4-Turbo on some tasks at lower cost, with fact-checking inference costing $0.31 on fine-tuned Mistral-7B-Instruct against $2.65 for zero-shot GPT-4-Turbo. Those were models of about seven billion parameters, not 70B.

TechnoLynx runs this comparison as the LLM Selection Pack. The engagement fine-tunes and serves one or more open-weight candidates, scores them against your hosted model on the same analyst queries, and reports cost per request and quality side by side, with the monthly figure for the volume share you are considering. It starts by agreeing the summarisation task, the quality metric and the target share with your analysts.

The comparison runs in this order:

1. Freeze the task: a representative sample of real analyst queries and source documents from your logs, plus the prompt templates in production today.
2. Pick candidates from the hardware and quantisation shortlist, then fine-tune them on representative summarisation data. The Laboratory-Scale AI authors found a 7B summarisation model reached 0.79 accuracy after 240 training samples, within 0.02 of its best. Fine-tuning data still has to be representative for that to transfer.
3. Serve each candidate on the stack you would deploy (vLLM, TensorRT-LLM or llama.cpp with GGUF), because each framework-model combination has its own performance characteristics.
4. Run the identical query set through the hosted model and every candidate, with matched inputs and sampling settings.
5. Score the outputs with a scoring method fixed in advance, and measure cost per request on each path.
6. Extrapolate to monthly cost at the target share, using the working shown below.

Benchmarking a local LLM on your own documents instead of generic scores

OpenAI’s evaluation guidance recommends task-specific evals that reflect real-world distributions, built partly by mining your own logs for cases. Anthropic’s guidance adds edge cases such as overly long inputs and cases where even humans struggle to agree. Where labelled examples are scarce, YourBench generates benchmarks directly from your own documents without manual annotation. When it replicated seven MMLU subsets, it preserved the original model ranking exactly (Spearman ρ = 1.00), and human annotators judged about 85% of its generated questions valid after filtering. For a fuller treatment of harness design, see our guide to building credible, data-specific evaluation harnesses.

How you score the outputs shapes the conclusion. Zheng et al. (2023) found that strong LLM judges agreed with human preferences more than 80% of the time, the same level at which humans agree with each other, but they also documented position, verbosity and self-enhancement biases. OpenAI advises validating an LLM judge against your own human labels before scaling it up. We would fix the scoring method and disclose it with the result, because swapping raters halfway through makes the comparison meaningless.

Worked example: turning per-request measurements into a monthly figure

The method follows the cost-per-million-tokens calculation in the FMwork paper: measured throughput gives the time to produce a unit of work, which is then multiplied by the hourly cost of the devices needed to reach that throughput. With your own measurements in place of the symbols:

• Hosted cost moved = monthly requests × share moved × measured hosted cost per request.
• Local cost per request = hourly cost of the devices needed ÷ requests served per hour at your measured throughput.
• Local monthly cost = monthly requests × share moved × local cost per request, multiplied by an operations factor for staff and monitoring.

One assumption that moves the answer sharply is utilisation. A rented GPU bills the same whether it is busy or idle, so at 10% utilisation the effective cost per token is 10× higher. The figures are only valid for the documents, prompts and fine-tuning data you tested, and they need re-running as your documents change.

How do we set pass/fail tolerances before comparing a local model against a hosted one?

Write the thresholds down before any results exist, per metric and on a named held-out set, so the comparison cannot be argued back to fit its outcome. Anthropic’s example of a good criterion shows the form. A summarisation equivalent would name the rubric score, the factual-error rate and the latency budget the local model must meet relative to the hosted baseline.

Tolerances also need statistics. The authors of “Adding Error Bars to Evals” (2024) show that when two models answer the same questions, a paired analysis of per-question differences reduces the standard error. That paired analysis helps tell a real difference from noise. Published tolerances vary with the purpose. An FP8 quantisation study across 75 architectures counted a workload as passing when it stayed within 1% relative loss of its FP32 baseline. A 2025 on-premises cost study treated benchmark scores within 20% of commercial models as parity. That 1% bar measured a quantised model against its own FP32 baseline. The looser bar is a procurement judgement, and we would only accept it if your own task scores show the gap does not matter to your users.

How do we know FP8 quantisation hasn’t degraded our model’s outputs?

Run the same held-out set through the full-precision and FP8 checkpoints under one identical harness, compare per-metric deltas against tolerances agreed in advance, and gate on task metrics rather than perplexity alone. We would not accept perplexity parity alone as proof that task scores held.

LynxBenchAI publishes how candidate devices run the public PyTorch stack, which informs the hardware side of an FP8 decision, not how your harness will score. The first step is to check its published results for the devices you are considering.

TechnoLynx builds this parity harness for your stack. It loads both checkpoints, runs them over your held-out set, computes the agreed metrics and is handed over for regression testing whenever checkpoints or quantisation settings change. It starts by fixing the downstream use case and the tolerance for each metric.

Which metrics to use depends on the task. For structured outputs (classification, extraction, code), exact match and task accuracy on held-out prompts should be the main gate. For open-ended generation, use task-specific scoring with a rubric or a validated judge, and keep perplexity as a secondary sanity check. We would not rely on exact match alone for generative tasks where several outputs are equally valid. For the distribution check, llama.cpp’s perplexity tool computes KL divergence between FP16 and quantised logits, where 0 means the distributions are identical. Symmetric percentiles in that output suggest quantisation is only adding noise, while much larger negative values suggest the model is actually getting worse.

For 70B specifically, the sources include a study of the whole Llama-3.1 family, based on more than 500,000 evaluations. It found FP8 weight-and-activation quantisation lossless at every scale, properly tuned INT8 within 1–3% degradation, and INT4 weight-only competitive with INT8. On cost, W4A16-INT gave the 8B and 70B models a 2–3× reduction in cost per query and 1.5–2× lower latency than BF16 in synchronous deployment, across all GPU types and latency-sensitive use cases tested. Averaged across academic benchmarks, every quantisation scheme recovered around 99% of the BF16 score. For asynchronous serving, W8A8-FP performed best on H100 and W8A8-INT on A100. Those are averages on public benchmarks, and the reason for running your own harness is to find out whether your task falls inside them.

Two caveats keep the result honest. The ELIB authors make the same point: GPU acceleration has to be tested on each platform and framework combination. Second, a single run hides variance. In the “Adding Error Bars to Evals” worked example, raising the samples per question from 1 to 10 cut the minimum detectable effect from 13.2% to 7.5%. Several seeds or samples per prompt are what make a 1% tolerance meaningful. The FP32 vs FP16 vs INT8 question works the same way: whatever the format, the harness and held-out set stay fixed, and only the checkpoint changes.

Does self-hosting a 70B model break even against API pricing at our volume?

Only if your sustained utilisation stays high. That makes break-even something you work out from measured throughput and your real traffic pattern, not from list prices. TechnoLynx turns that measurement into a decision through the LLM Selection Pack: cost per unit of work and latency for each hosted API and the self-hosted candidate at your expected volume. It starts by defining your production volume, latency targets and cost metric.

Cost and throughput across hosted APIs and the open-weight model

Three effects dominate the comparison. The first is utilisation. LeanLM calculates that, against a hosted open-weight endpoint, a self-hosted H100 breaks even on raw compute at about 17% sustained utilisation. Once a 3–5× operations factor is added, it needs roughly 50–80% just to tie. The second is batching. In NVIDIA’s benchmark cited by LeanLM, a dense 70B on two H100s in FP8 produced 51.7 tokens/s at concurrency 1 and 920 at concurrency 250, so break-even maths built on single-stream numbers is wrong from the start. The third is the time horizon. A 2025 cost-benefit study put break-even for medium models such as Llama-3.3-70B at 3.8 to 34 months, depending on the provider compared, on $15k–$30k of dual-A100 hardware for enterprises processing 10–50M tokens a month. That study counts only electricity as the recurring cost, so staffing would lengthen those periods.

Precision also moves the cost. A 2025 theoretical model of inference economics for Llama 3 70B on H100s estimates $0.23 per million tokens with 4-bit weights, $0.37 with 8-bit and $0.70 with 16-bit. Those are modelled optima, not measurements of your deployment.

Before switching models, profile the serving path you already have. Cost and latency problems in production AI rarely need a new model: batching, caching, routing, quantisation, kernel work, serving configuration and runtime choice sit below the model and are tunable independently. Replacing the model is the most expensive lever, and it often leaves the workload economics unchanged. The same applies after a switch: single-stream throughput understates what a batched 70B serves, so judge cost under realistic concurrency. Self-hosting break-even is sensitive to hidden operating costs, and vendor prices change, so the analysis needs a date and a re-check schedule.

Data-use terms are a separate gate. Whether a vendor contract permits evaluation or production use of de-identified, PHI-adjacent data depends on that vendor’s own terms, which need checking directly. We would treat it as pass/fail before any cost comparison counts, with your legal team making the call.

Legacy prompt patterns versus structured output and tool calling

Whether older few-shot or chain-of-thought templates still earn their tokens once a model supports JSON mode or function calling is answered the same way: run both paths over your own eval set and compare. OpenAI lists function-call accuracy and executable checks among the quantitative eval types suited to automated regression testing. For each path, we would count malformed outputs, task accuracy, tokens per request and latency, then keep the old template only where it measurably beats the structured path on your tasks. Any failure-rate figure you rely on has to come from your own run.

Who can run a local-versus-hosted LLM comparison for us?

TechnoLynx shortlists 70B hardware and quantisation, fine-tunes and serves the candidates, builds the parity and evaluation harness on your documents, and reports cost per request and quality against your hosted model, with a monthly projection for the share you plan to move. On build in-house versus bring in a specialist: if your team already maintains an evaluation harness and a production serving stack, the work can stay in-house. If not, the risk is a comparison nobody trusts. When choosing a partner, ask whether they will fix the metrics and tolerances before running anything and hand over the harness afterwards. The LLM Selection Pack is the engagement for this, and it sits within TechnoLynx’s Generative & Agentic AI R&D work.

For a team deciding which card to buy without running a model, LynxBenchAI’s published device results are the better starting point. The pack does not cover legal review of vendor data-use terms. A team with mature MLOps can follow the harness approach in our evaluation harness guide and bring in help only for the serving or fine-tuning step.

What a passing comparison still leaves open

A comparison that passes tells you the local model met your tolerances on the documents you tested, on the stack you tested, at the volume you assumed. It does not cover what happens when any of those change. A different GPU, platform or framework configuration can each move quality or performance. That is why the harness matters more than the one-off result, and why it should run as a regression test on every checkpoint or configuration change. If the same harness keeps passing on fresh samples from your logs, you can trust the decision. If you cannot re-run it, the decision has an expiry date you cannot see.

FAQ

Is FP8 or INT8 better?

In a study across 75 network architectures, FP8 passed a 1% accuracy criterion on 92.64% of workloads against 65.87% for INT8. For Llama-3.1, FP8 was lossless while tuned INT8 lost 1–3%. Which format is most cost-effective depends on the GPU: for asynchronous serving, W8A8-INT suited A100 and W8A8-FP suited H100.

What are BLEU and ROUGE scores used for in LLM evaluation?

Anthropic lists BLEU among task-specific quantitative metrics. OpenAI lists ROUGE/BLEU scoring among quantitative evals, and one quantisation study used ROUGE to compare a quantised model’s outputs with its full-precision version.