“The model shall be robust to adverse weather.” That sentence has passed more requirements reviews than any single line of a perception spec deserves. It reads like a specification. It behaves like a wish. The problem is not the intent — it is the shape. A robustness requirement written as one line, validated against one benchmark, with one acceptance threshold, cannot bound the behaviour of a perception model across the conditions where it actually gets tested: rare fog, low sun angles, a camera bracket that has drifted two degrees since the last calibration. The model passes the requirements document and then fails on the road. That gap has a name in how we work with automotive teams — we call the practice that closes it SpecForge: forging a robustness claim into a structured set of per-scenario-class acceptance criteria before validation begins, each class carrying its own operating conditions, evidence requirement, and pass threshold. This article is about why the single-line spec fails and what replaces it. It is not a benchmark writeup and it is not a monitoring playbook — it is the step before both, where you decide what “robust” is allowed to mean and how you will prove it. What does SpecForge mean in practice? SpecForge is a decomposition discipline. You start from a robustness claim that a stakeholder cares about — pedestrian detection holds in degraded visibility, say — and you break it into scenario classes narrow enough that a single acceptance threshold is defensible within each one. A scenario class is a bounded slice of the operating domain: “pedestrian detection, dusk, backlit, dry road, nominal camera mounting” is a class. “Adverse weather” is not — it is a folder of classes pretending to be one. The naive spec collapses the folder into a line. SpecForge keeps the folder open and writes a contract for each item in it. Concretely, forging the spec produces three artifacts that did not exist before: A scenario-class inventory — the enumerated list of operating conditions the model is claimed to handle, each with explicit boundaries. A per-class contract — for every class, the operating conditions that define it, the evidence that will demonstrate performance, and the numeric threshold that counts as a pass. A coverage map — an honest statement of which classes are specified, which are known-unspecified, and which are out of the operational design domain entirely. The last one matters more than teams expect. A specification is not only a list of promises; it is a list of the promises you are deliberately not making. Writing down the unspecified region is what stops a reviewer — or a regulator — from assuming a blank means “handled.” Why a single robustness line fails where scenario classes succeed The failure mechanism is the long tail, and it is structural rather than a matter of writing more carefully. When you validate one robustness line against one benchmark, you get a single aggregate number — say, mean average precision across a mixed validation set. That number is dominated by the common conditions, because the common conditions dominate the dataset. Rare fog might be 0.4% of frames. A model that fails badly on fog and well on everything else can still post a strong headline score, because the fog frames are numerically drowned out. This is an observed pattern across the perception-validation engagements we run: the aggregate metric and the tail-class metric diverge, and the divergence is exactly where post-release surprises come from. We have written more on the perception failure modes that survive benchmarks — SpecForge is the upstream practice that would have caught them. A per-scenario-class specification inverts the arithmetic. You are no longer averaging fog into everything else; you are asking whether the fog class, on its own, clears its own threshold with its own evidence. The tail stops hiding inside the aggregate. It gets its own line, its own pass condition, and its own place to fail loudly during validation instead of quietly in the field. There is a second, quieter benefit. A single headline score invites re-litigation — every reviewer forms a private opinion about whether 0.87 mAP is “good enough,” and the review meeting becomes a negotiation. Per-class contracts pre-agree the thresholds before validation runs. The reviewer’s job shifts from judging a number to checking that pre-agreed evidence exists. That is why teams that forge the spec first tend to see shorter validation pass-through time — the argument happened once, in the spec, instead of every release. How do you decompose “robust to adverse weather” into testable criteria? Start from the axes that actually change model behaviour, then take the cross-product and prune. For a camera-based detector, the axes that move the needle are usually illumination (bright, dusk, night, backlit, tunnel-transition), precipitation and its optical effect (clear, light rain, heavy rain, spray, fog by visibility band), surface and geometry (dry, wet-reflective, low sun on wet road), and the sensor’s own state (nominal mounting, mounting drift within tolerance, lens contamination). The cross-product is large; most of it you will never specify, and that is fine. You prune to the classes that carry safety weight or that historically produce failures. Here is a worked fragment for one robustness claim, with explicit assumptions stated so the thresholds are interpretable rather than magic numbers. Worked example: decomposing “pedestrian detection in degraded visibility” Assumptions: front camera, ODD limited to structured roads ≤ 80 km/h; thresholds illustrative, not benchmarked; evidence measured on a curated per-class validation set with a stated minimum frame count per class. Scenario class Operating conditions Evidence requirement Pass threshold (illustrative) Dusk, backlit, dry Sun 0–10° elevation, dry surface, nominal mounting ≥ 2,000 labelled frames, per-class recall reported with CI Pedestrian recall ≥ 0.95 at fixed FP rate Fog, moderate Meteorological visibility 100–200 m, daytime ≥ 1,500 frames, recall vs range curve Recall ≥ 0.90 for targets ≤ 40 m Night, wet-reflective No sun, wet surface, streetlight only ≥ 2,000 frames, recall + localisation error Recall ≥ 0.92, lateral error ≤ 0.3 m Nominal, mounting drift Clear day, extrinsic yaw drift ≤ 2° ≥ 1,000 frames at drift boundary No recall drop > 3% vs calibrated baseline Each row is a contract you can validate independently and fail independently. The fog row failing does not blur into the night row passing. And the last row — mounting drift — is the one most single-line specs forget, which is the whole reason the next two sections exist. What belongs in each scenario-class contract? Three fields, and all three are load-bearing. Operating conditions define the class boundary precisely enough that someone else could assemble the validation data for it without asking you. “Fog” is not a boundary; “meteorological visibility 100–200 m, daytime, dry road” is. Vague boundaries are where scope silently expands — a reviewer assumes the fog contract covers 50-metre fog, the data curator sampled 150-metre fog, and nobody notices until the field data disagrees. Evidence requirement states what measurement demonstrates performance and how much of it is enough. This is where you specify the metric and, critically, its denominator. A recall figure computed on 30 fog frames is not evidence; it is noise with a decimal point. The contract names the minimum sample, the metric, and the reporting form (point estimate, confidence interval, or a curve against range or distance). Choosing the metric well is its own discipline — we cover what each performance metric actually proves for perception, because a spec that demands the wrong metric will pass a model that should have failed. Pass threshold is the pre-agreed number. It is set during spec forging, with stakeholders in the room, and it is frozen before validation runs. Freezing it is the point. A threshold negotiated after seeing the result is not a threshold; it is a rationalisation. How do sensor mounting and calibration drift get written in? This is the class that separates a specification written by people who have deployed perception from one written by people who have only trained it. A perception model is calibrated against a specific camera geometry — extrinsics describing where the camera sits and how it points. In production that geometry drifts: thermal cycling, vibration, a service technician who reseats a bracket. The model was validated at nominal mounting and then runs at nominal-plus-drift, and its behaviour at the drift boundary is often unspecified because the spec treated calibration as a fixed precondition rather than a variable. We have written separately on why calibration drift breaks a tracking model and on why extrinsic calibration belongs in your safety evidence — the specification is where that concern becomes testable. The way to write it in is to make mounting state an explicit axis of the scenario-class inventory, exactly like weather. You define the drift tolerance the vehicle is expected to hold — a yaw and pitch envelope — and you create classes at the boundary of that envelope, not only at the nominal centre. The contract then reads: at extrinsic drift up to the specified tolerance, per-class recall must not degrade beyond a stated margin against the calibrated baseline. That single move converts calibration from an implicit assumption into a validated, evidenced claim. It also tells the monitoring system what to watch for in the field, because the spec has now named the failure it is guarding against. When is a scenario-class specification complete enough to gate a release? Never fully, and that is the honest answer — the operational domain is larger than any inventory. Completeness is not “every possible condition is specified”; it is “every condition that carries safety weight is either specified or explicitly declared out of scope, and the boundary between them is written down.” Use this rubric to decide whether the spec is release-gating. Release-gating readiness checklist Every safety-relevant scenario class has a contract with all three fields populated (conditions, evidence, threshold). Each contract’s evidence requirement names a minimum sample size, not just a metric. Mounting/calibration drift appears as an explicit axis, with at least one boundary class. The coverage map lists known-unspecified classes — the blanks are deliberate, not accidental. Thresholds were frozen before validation ran (check the timestamps, not the intent). The long-tail classes have their own pass conditions, separable from the aggregate. A reviewer can sign off by checking evidence against contracts, without re-deriving thresholds. When every box is checked, the specification is doing the job a specification exists to do: it lets a reviewer accept per-class evidence against a pre-agreed contract instead of arguing about a headline score. That is the artifact that feeds the [production AI monitoring harness](Production AI Monitoring Harness) — the spec defines what the harness measures against in the field, and the per-class contracts become the audit’s reference. FAQ What does working with specforge involve in practice? SpecForge is a decomposition discipline: it takes a robustness claim a stakeholder cares about and breaks it into scenario classes narrow enough that a single acceptance threshold is defensible within each. In practice it produces three artifacts — a scenario-class inventory, a per-class contract (operating conditions, evidence requirement, pass threshold), and a coverage map that states honestly which classes are unspecified. It is the step before benchmarking and monitoring, not a replacement for either. Why does a single robustness requirement line fail where a per-scenario-class specification succeeds? A single line validated against one benchmark yields one aggregate number dominated by common conditions, so a model that fails on rare fog can still post a strong headline score because the fog frames are numerically drowned out. A per-scenario-class specification asks whether each class clears its own threshold with its own evidence, so the long tail gets its own line and fails loudly during validation instead of quietly in the field. How do you decompose “robust to adverse weather” into testable scenario-class acceptance criteria? Start from the axes that actually change model behaviour — illumination, precipitation and its optical effect, surface geometry, and sensor state — then take the cross-product and prune to the classes that carry safety weight or historically produce failures. Each resulting class gets explicit operating-condition boundaries, a stated evidence requirement, and a frozen pass threshold, so it can be validated and failed independently of the others. What belongs in each scenario-class contract — operating conditions, evidence requirement, and pass threshold? Operating conditions define the class boundary precisely enough that someone else could assemble the validation data without asking you. The evidence requirement names the metric, its minimum sample size, and its reporting form (point estimate, confidence interval, or curve). The pass threshold is the pre-agreed number, frozen before validation runs — a threshold negotiated after seeing the result is a rationalisation, not a threshold. How does a robustness specification connect to the validation evidence pack a release reviewer accepts? The specification defines the contracts the evidence pack is assembled against: each per-class contract becomes a line the reviewer checks evidence against, rather than a number to re-litigate. Because thresholds were frozen during spec forging, sign-off shifts from judging a score to confirming that the pre-agreed per-class evidence exists, which shortens validation pass-through time. How does sensor-mounting variance and calibration drift get written into the specification rather than left implicit? You make mounting state an explicit axis of the scenario-class inventory, just like weather, defining the drift tolerance the vehicle is expected to hold and creating classes at the boundary of that envelope rather than only at nominal. The contract then requires that, at drift up to the specified tolerance, per-class performance not degrade beyond a stated margin against the calibrated baseline — turning an implicit precondition into a validated, evidenced claim. When is a scenario-class specification complete enough to gate a release against? Completeness is never total — the operational domain is larger than any inventory — so the standard is that every safety-relevant class is either specified or explicitly declared out of scope, with the boundary written down. The release-gating checklist confirms each contract has all three fields, evidence names a sample size, mounting drift appears as an axis, the coverage map lists deliberate blanks, and thresholds were frozen before validation ran. The line you decide not to write The hard part of a robustness specification is not enumerating what the model must handle. It is deciding, deliberately and in writing, what it is allowed not to handle — and then making sure the classes you did specify are narrow enough that a single number can honestly gate them. A benchmark-blind release passes the document and fails the road because the document averaged the tail away. SpecForge is the practice of refusing that average: forging the per-scenario-class contracts before validation begins so the tail has somewhere to fail on your terms, in your test harness, instead of a stranger’s. The remaining question for any team about to write “shall be robust” is simpler than it looks — robust to which class, measured how, and against what threshold you were willing to freeze before you saw the result.
