A procurement committee writes down what a model must do — tolerable failure modes, a latency ceiling, a cost-per-request limit — and hands the document to an engineer. That document is not runnable. The gap between a written requirement and a number that answers it is where most eval work quietly goes wrong, and it is the gap a spec processor exists to close. The naive fix is to let one engineer read the requirements, form a mental model of what matters, and pick a handful of metrics off a vendor leaderboard. It looks efficient. It also loses the one thing that makes an eval defensible: the link between “the committee asked for X” and “this number answers X.” A spec processor is the repeatable step that parses a written eval spec into a concrete, runnable metric configuration — one that maps each stated requirement to a specific measurement, and keeps that mapping intact all the way to the report. What a spec processor actually does Think of it as a compiler for eval intent. On one side you have prose: a paragraph saying the model “must not hallucinate customer account balances,” a line item capping p95 latency at 800ms, a budget line that no request may exceed a fraction of a cent. On the other side you have a machine-executable configuration: which dataset to run, which scoring function to apply, under which serving conditions, and which threshold turns a raw number into a pass or fail. The processor’s job is to walk from the first to the second without dropping the trail. Every metric in the output configuration carries a back-reference to the requirement that justified it. When the eval report lands on the committee’s desk, each number can be read as a direct answer to a question they actually wrote down — not as a plausible-looking figure an engineer chose because it was available. We see the same pattern regardless of model or vendor: the moment the requirement-to-metric link is implicit rather than recorded, the eval stops being an argument and becomes a collection of numbers. Numbers without provenance invite a second opinion, and a second opinion means a second eval round. How does a spec processor work in practice? It runs as a front-end stage before any model is touched. The flow, in the configurations we have set up, tends to look like this: Parse. Read the written spec and extract discrete, testable requirements — each with a subject (what is measured), a constraint (the threshold), and a condition (under what serving setup). Resolve. For each requirement, select the measurement that answers it: a dataset, a scoring rule, and a run condition. A latency ceiling resolves to a load profile plus a percentile; a factuality requirement resolves to a grounded QA set plus a hallucination-scoring rule. Bind. Attach the threshold to the metric so the raw result yields a verdict, not just a value. Emit. Produce a configuration the validation harness can run directly, with each entry carrying its requirement back-reference. The output is not a document for humans to interpret. It is the input to the [production AI monitoring harness](Production AI Monitoring Harness) that actually runs the metrics — the processor is the front end, the harness is the engine. Why manual metric selection drifts The failure class here is requirement-to-metric drift, and it is structural, not a matter of engineer skill. A written spec contains constraints that do not map one-to-one onto public benchmarks. “Must not confidently state a wrong account balance” is a specific, workload-shaped failure mode; the nearest leaderboard number might be an aggregate factuality score on a generic QA set. Picking that score feels reasonable. It also silently substitutes a different question for the one the committee asked. Once that substitution happens without a record, nobody can audit it later. When a stakeholder asks “which number tells me the model won’t invent balances?”, the honest answer is “none of them directly — we used a proxy.” That is the conversation that triggers a re-eval. As with the mapping described in how an evaluation spec links task, dataset, scoring, and run conditions, the coupling between requirement and measurement is the thing that has to survive; a spec processor is the mechanism that keeps it from decaying into an engineer’s memory. There is a second, quieter cost. Manual setup effort scales with every new candidate model. Evaluate three vendors and someone re-does the metric selection three times, each with its own small drift. A processed spec produces one canonical configuration that runs identically across candidates — which is also what makes the cost-per-request comparisons fair across serving configs rather than an artifact of who set up which run. Manual selection vs. a spec processor The difference is easiest to see as a side-by-side on the properties a procurement committee cares about. Property Manual metric selection Spec processor Requirement-to-metric link Lives in an engineer’s head Recorded per metric, back-referenced to the spec Reproducibility across candidates Re-done per model, drifts each time One canonical config, identical runs Auditability “Trust the engineer’s judgement” Each number traces to a stated question Coverage of stated failure modes Implicit, easy to miss one Explicit checklist derived from the spec Time to a second candidate eval Scales with headcount Marginal — rerun the same config Re-eval risk High — proxies surface under scrutiny Low — mapping defends itself This is a observed-pattern comparison drawn from procurement evals we have set up, not a benchmarked measurement of two teams racing each other. The point is not a stopwatch figure; it is where the two approaches diverge under scrutiny, which is exactly the moment a purchase decision gets challenged. What belongs in an eval spec A spec processor is only as good as the spec it parses. The most common reason a processor produces leaderboard noise instead of a defensible metric set is that the written spec never stated a testable constraint in the first place. A usable eval spec needs, for each requirement: A subject — the specific behaviour being measured, in workload terms (“account-balance factuality”), not category terms (“quality”). A constraint — the threshold that separates acceptable from unacceptable, expressed as a number where possible. A condition — the serving setup the measurement must run under, because the same model gives different latency and cost figures under different serving configs and runtime choices. A failure consequence — what breaks in production if this requirement is missed, which lets the processor rank severity when trade-offs collide. Requirements missing a constraint or condition can still be parsed, but they resolve to weaker metrics. “The model should be fast” becomes an arbitrary percentile choice; “p95 latency ≤ 800ms at 40 concurrent requests” resolves cleanly. Writing the spec well is the highest-leverage thing a committee can do to make the downstream eval defensible. Where the spec processor sits — and where it stops The processor is a front end. It does not run models, it does not measure anything, and it does not decide the purchase. It converts intent into a runnable configuration and hands that configuration to the harness. Keeping this boundary sharp matters: a spec processor that starts editorializing about which model is better has left its lane. It also does not replace human judgement everywhere. Some requirements are genuinely subjective — “responses should feel on-brand” — and resolve to a human-in-the-loop rubric, not an automated score. The processor’s job there is to flag the requirement as requiring manual judgement and to record that decision, not to fake a number. This is where a purely leaderboard-driven approach fails most visibly, and why Chatbot Arena and similar public rankings can’t replace a spec-driven eval: they answer a general preference question, not the specific one the committee wrote. How do you check the processed metric set covers the failure modes? Coverage is the validation step people skip. The processor produces a mapping; you still have to prove the mapping is complete. A workable check: Reverse-trace. For every failure mode named in the spec, confirm at least one metric back-references it. An unmatched failure mode is a coverage hole. Forward-trace. For every metric in the config, confirm it back-references a real requirement. An unmatched metric is leaderboard noise that crept in. Threshold sanity. Confirm each metric’s bound actually distinguishes pass from fail on a known-bad example, not just a number that everything passes. When both traces close and the thresholds bite, the metric set is defensible — which is the same traceability a governance evidence pack needs to defend a procurement choice after the fact. The requirement-to-metric mapping the processor produces is not just an eval convenience; it is the audit trail. FAQ How does spec processor work? A spec processor is a repeatable front-end stage that reads a written eval spec and emits a runnable metric configuration. In practice it parses each requirement into a subject, constraint, and condition, resolves each to a specific measurement, binds a threshold so the result yields a verdict, and emits a config the validation harness runs directly. It means the eval is defined once, executably, instead of re-improvised per engineer. What inputs does a spec processor take, and what runnable metric configuration does it produce? It takes the written eval requirements — the committee’s document of tolerable failure modes, latency ceilings, and cost limits. It produces a machine-executable configuration listing which dataset to run, which scoring function to apply, under which serving conditions, and which threshold turns a raw number into a pass or fail. Each entry carries a back-reference to the requirement that justified it. How does a spec processor keep each reported metric traceable back to a specific procurement requirement? Every metric in the output configuration carries a recorded back-reference to the requirement it answers, so the link is stored in the config rather than in an engineer’s memory. When the report lands, each number reads as a direct answer to a question the committee wrote down. That stored mapping is what survives scrutiny and prevents the eval from becoming a collection of unattributed numbers. What belongs in an eval spec so the processor can turn it into a defensible metric set rather than leaderboard noise? Each requirement needs a subject in workload terms, a constraint expressed as a number where possible, a condition naming the serving setup, and a failure consequence. Requirements missing a constraint or condition still parse but resolve to weaker, arbitrary metrics. A well-written spec is the highest-leverage thing a committee can do to make the downstream eval defensible. Where does a spec processor sit relative to the validation pack that actually runs the metrics? The processor is the front end; the validation harness is the engine. The processor converts written intent into a runnable configuration and hands it off — it does not run models, measure anything, or decide the purchase. Keeping that boundary sharp is deliberate: a processor that starts ranking models has left its lane. What does a spec processor not handle, and which requirements still need manual judgement? It does not run models, measure performance, or make the purchase decision, and it cannot automate genuinely subjective requirements like brand tone. For those, it resolves to a human-in-the-loop rubric and flags the requirement as needing manual judgement rather than faking a score. Recording that decision is part of the trace. How do you validate that the processed metric set actually covers the failure modes the committee asked about? Run a reverse-trace confirming every named failure mode has at least one metric back-referencing it, a forward-trace confirming every metric maps to a real requirement, and a threshold sanity check confirming each bound actually distinguishes a known-bad example from a passing one. Unmatched failure modes are coverage holes; unmatched metrics are noise that crept in. When both traces close and thresholds bite, the set is defensible. The question that decides the round The test of a spec processor is not how clever the parsing is. It is whether, when a stakeholder points at any number in the final report and asks “which requirement does this answer?”, the answer is already written down. Manual selection can produce good numbers; it cannot reliably produce that answer, and the missing answer is what turns one eval into two. If your current eval setup can’t reverse-trace every reported metric to a stated procurement requirement, that gap — not the model — is what your next re-eval round is really about, and the [production AI monitoring harness](Production AI Monitoring Harness) is where that trace gets enforced.
