“We’ll go LLM-agnostic” usually means something narrower than the people saying it believe. In most procurement conversations it is treated as a checkbox: pick whatever model tops the leaderboard this quarter, wire it in, and assume a swap later is trivial. Then the day comes — a new model leads, a provider raises its price, an endpoint gets deprecated — and the “swap” turns out to be a multi-week integration project. The stack was never agnostic. It was hard-wired to one vendor’s API shape, prompt quirks, and output format, with a leaderboard reading standing in for the evaluation that should have been done. LLM-agnostic architecture is not a property you claim. It is a design you build, and its whole point is narrow and specific: being LLM-agnostic means you can re-decide on evidence when conditions change — it does not mean every model performs equally on your task. Those two ideas get conflated constantly, and the conflation is where the money leaks. How does LLM-agnostic work in practice? The naive mental model is that “agnostic” describes the models — as if any of them would do, interchangeably. That is not it. Different models genuinely differ on your task: a model that leads on a general chat leaderboard may underperform on your classification distribution, your extraction schema, or your domain vocabulary. Treating them as interchangeable is how teams end up disappointed by a “better” model that scores worse on the work they actually do. What “agnostic” actually describes is your architecture, not the models. An agnostic system is one where the identity of the model behind an abstraction boundary is a configuration value, not a structural assumption baked into a dozen call sites. The provider can change; the surrounding code, prompts, and evaluation harness do not. When the leaderboard reshuffles, an agnostic team does not rebuild — they re-run a task-specific evaluation against the new candidate and either switch or stay, on evidence. This is the same distinction we draw in why a leaderboard number isn’t your number: the public ranking measures a different distribution than yours. Agnostic architecture is what lets you cheaply substitute the leaderboard’s answer with your own. What does a model-independent architecture actually consist of? Three components, and all three have to exist for the property to hold. Most stacks that call themselves agnostic have built one and skipped the other two. The abstraction layer sits over the provider API. It normalises the request and response shape so the rest of your system talks to a stable internal interface rather than directly to OpenAI’s, Anthropic’s, or a self-hosted model’s specific schema. Frameworks like LiteLLM, LangChain’s model interface, or a thin in-house adapter all serve this role. The abstraction is not glamorous, but it is what turns provider identity into a config value. Without it, provider-specific calls leak into business logic and every swap touches everything. The prompt harness keeps prompts versioned, parameterised, and decoupled from any single vendor’s formatting idiosyncrasies. Models differ in how they respond to system-message placement, JSON-mode reliability, tool-call formats, and few-shot structure. A harness that treats prompts as first-class, tested artifacts — rather than string literals scattered through the code — is what lets you retune for a new model without archaeology. In practice this means prompt templates, per-model overrides where genuinely needed, and a way to diff behaviour across versions. The evaluation harness is the component that makes a swap safe rather than merely possible. It runs your candidate models against your prompt distribution and scores them on the metrics that matter for your task — accuracy, recall on the classes you care about, latency, cost per decision. Reading recall correctly here is its own discipline; we cover it in reading recall in an LLM evaluation pack. The eval harness is what you re-run when conditions change. If it exists and stays warm, a model change is hours of work. If it doesn’t, you are back to guessing. Quick check: do you actually have an agnostic architecture? Component Present if… Failure signal Abstraction layer Provider is a config value; no vendor-specific calls in business logic Swapping providers requires editing many call sites Prompt harness Prompts are versioned, tested artifacts with per-model handling Prompts are string literals tuned once for one vendor Eval harness You can re-score a new model on your task distribution in hours You’d rely on a leaderboard reading to justify a swap Warm shortlist 2–3 candidate models kept eval-ready, not just the incumbent Only the current provider has ever been evaluated If any row shows the failure signal, the “agnostic” label is aspirational. The property is only as strong as the weakest of the four rows. Why doesn’t being LLM-agnostic mean every model performs equally? This is the most common misreading, and it is worth stating plainly because it changes what you build. Agnostic architecture removes the integration cost of changing models. It does nothing to the performance differences between models on your specific task — those are real, they are large, and they are exactly what a general-purpose leaderboard cannot tell you. A model that wins on Chatbot Arena’s pairwise-preference ranking is winning on aggregate human preference over open-ended prompts. Your task might be extracting structured fields from clinical notes, classifying support tickets, or summarising regulatory filings. The correlation between arena Elo and your task accuracy is often weak, sometimes inverted. In configurations we have evaluated, a model that ranked lower on public leaderboards outperformed the top-ranked one on a narrow classification task, because the task rewarded consistency and schema-adherence rather than conversational fluency (observed pattern across evaluation engagements; not a benchmarked rate). So the point of agnostic design is not to make model choice irrelevant. It is the opposite: it makes model choice cheap to re-examine, so you can act on the task-specific performance differences that matter instead of committing to a leaderboard proxy. The architecture buys you the option to run the evaluation. The evaluation is still the thing that decides. If leaderboard rank doesn’t predict task accuracy — and for classification it frequently doesn’t, as we discuss in when leaderboard rank doesn’t predict task accuracy — then the only defensible basis for a swap is your own measurement. How does agnostic design change switching cost? Switching cost is the measurable outcome, so it is worth being concrete about what the two architectures cost on the day a change is forced. Worked comparison: the day the leaderboard leader shifts Assume a new model appears that claims a meaningful accuracy improvement, and your incumbent provider simultaneously announces a price increase. What happens next depends entirely on whether the three components above exist. Step Hard-wired stack Agnostic stack Discover the change Same for both Same for both Add the candidate model Rewrite provider-specific call sites Add a config entry behind the abstraction layer Re-tune prompts Rewrite scattered prompt strings, no version control Adjust harness templates, diff against baseline Evaluate on your task No harness exists — build one or guess from the leaderboard Re-run the existing eval harness on your distribution Decide and switch Multi-week integration and QA cycle Config swap once the eval clears Elapsed effort Weeks Hours to a day The delta between those two columns is the switching cost, and it is the honest ROI of agnostic design: the difference between re-running an existing task-specific eval harness and re-integrating a hard-wired dependency. It also caps pricing exposure. A warm shortlist of two or three continuously eval-ready models means no single provider can move price against you without you having a measured alternative ready to substitute — you are not committing cost-per-decision to one vendor by default (observed pattern; the size of the saving depends on your call volume and provider spread). What does a procurement committee gain that a leaderboard ranking cannot give? A leaderboard ranking gives a committee a snapshot of aggregate preference on a general task at one moment. It cannot tell them how a model performs on their workload, it cannot be re-run against their data, and it goes stale the moment the rankings reshuffle. It is a reasonable way to build a shortlist. It is not a decision. What an agnostic architecture gives a committee is a repeatable decision process. When someone asks “why this model?”, the answer is not “it was top of the leaderboard when we chose” — an answer that ages badly and survives no audit. The answer is “it won our task-specific evaluation against these candidates on these metrics, and here is the harness that will re-decide when conditions change.” That reusability is the connection to a procurement evidence pack: the same abstraction layer and eval harness that make a swap a config change are what make the evidence pack reusable across candidate models, so a leaderboard-driven shortlist gets re-scored on task-specific accuracy without re-engineering. The methodology for actually running that task-specific evaluation — the part that makes the swap safe — is a procurement-eval discipline in its own right, and it is where an agnostic architecture stops and the evidence work begins. The Bayesian view is useful here: a leaderboard is a prior, your evaluation is the evidence that updates it, and the agnostic architecture is what makes updating cheap. We work through that framing in turning new evidence into a defensible model choice. FAQ What does working with LLM-agnostic involve in practice? LLM-agnostic describes your architecture, not the models. It means the model behind an abstraction boundary is a configuration value rather than a structural assumption baked into your code. In practice, when the leaderboard reshuffles or a price moves, you re-run a task-specific evaluation against a new candidate and switch on evidence, rather than rebuilding the integration. What does a model-independent architecture actually consist of — abstraction layer, prompt harness, and eval harness? Three components, all of which must exist. The abstraction layer normalises the provider API so provider identity becomes a config value. The prompt harness keeps prompts versioned and decoupled from any one vendor’s formatting quirks. The eval harness scores candidate models on your task distribution and is what makes a swap safe. Missing any one leaves the “agnostic” label aspirational. Why doesn’t being LLM-agnostic mean every model performs equally on your task? Agnostic architecture removes the integration cost of changing models; it does nothing to the real performance differences between models on your specific task. A model that leads a general chat leaderboard may score worse on your classification or extraction work. The point of agnostic design is to make re-examining model choice cheap, so you act on task-specific performance rather than a leaderboard proxy. How does an agnostic design change switching cost when a leaderboard leader shifts or a provider deprecates an endpoint? It turns a multi-week integration project into a config swap plus an eval re-run. The hard-wired stack must rewrite call sites, re-tune scattered prompts, and build an evaluation from scratch; the agnostic stack adds a config entry and re-runs its existing harness. The measurable delta is the switching cost, and a warm shortlist of eval-ready models also caps vendor pricing exposure. What does a procurement committee gain from an agnostic architecture that a leaderboard ranking cannot give them? A repeatable, auditable decision process. A leaderboard is a stale one-moment snapshot of aggregate preference that cannot be re-run against their data. An agnostic architecture lets the committee answer “why this model?” with “it won our task-specific evaluation against these candidates on these metrics, and here is the harness that re-decides when conditions change.” Where does LLM-agnostic design end and the task-specific procurement evidence pack begin? The agnostic architecture supplies the reusable machinery — the abstraction layer, prompt harness, and eval harness — that lets any candidate be scored without re-engineering. The evidence pack is the output of actually running that task-specific evaluation and documenting the result. The architecture makes the evaluation cheap and repeatable; the evaluation, run through a procurement-eval methodology, is what produces the decision-grade evidence. The question worth carrying into the next procurement review The honest test of an agnostic architecture is not whether you can name the abstraction library you use. It is this: when your leading model changes next quarter — and it will — can you re-decide in an afternoon on evidence drawn from your own task, or are you looking at a migration project and a leaderboard screenshot? If it’s the latter, the label was doing work the architecture never did. The gap between those two answers is exactly the switching cost you are quietly carrying, and it is the first thing worth measuring before the swap is forced on you.
