A model approval holds up when someone who was not in the room can check every part of it. That takes four things. The first is one typed model card schema that the pipeline fills in and the registry enforces. The second is a documented comparison of every candidate, rejected ones included, against the current production model. The third is a promotion gate that fails closed and has been shown to block a known-bad checkpoint. The fourth is a fairness metric and threshold that the governance committee owns in versioned policy. This is model risk governance treated as engineering: the approval evidence is a deliverable the pipeline produces, not a memo someone signs. An auditor, a buyer or a model-risk committee has to be able to read that evidence, defend it and sign against it.

A hand-written card costs its author time today and helps a reviewer a year from now, so it loses to the deadline and then drifts out of date. A study of 32,111 public model cards on Hugging Face (Liang et al., 2024) found the Evaluation section filled in on 15.4% of cards and Limitations on 17.4%, while Training was filled in on 74.3%. The TMLS governance review notes those two sections are the ones a risk reviewer reads first. The MLflow project’s guidance on shared registries says it bluntly: a registry only governs what it actually gates.

What must every model card contain before a promotion counts as documented?

A promotion counts as documented when two things are true. The card passes one central schema whose required fields cover intended and out-of-scope use, data provenance, disaggregated evaluation metrics, known limitations, risk tier, and a named approver with a timestamp. And the registry refuses the stage transition when any of those fields is missing.

TechnoLynx does this work under AI Governance and Trust. The engagement starts with an inventory of the metadata each registered model already carries, measured against a target common schema. TechnoLynx then implements that schema as required fields in the registry’s approval gate, including a backfill mapping for models that are already registered.

Today the structure of model cards is inconsistent. The AI Transparency Atlas’s analysis of its sample of model cards (2025) found 947 distinct section names, and usage information alone appeared under 97 of them. The same authors propose a core schema of 20 to 30 essential fields that every high-impact model should report. One model registry governance guide makes model ID, version, owner, training data snapshot ID, commit hash, metrics and risk tier required fields. It also recommends a model signature, and makes the signature mandatory for anything customer-facing.

The required fields and who signs each one

Field	What it must hold	Owner
Identity and lineage	Model ID, version, commit hash, training data snapshot ID	Data science
Intended and out-of-scope use	What the model is for, and explicitly what it must not be used for	Data science drafts, regulatory affairs confirms
Disaggregated evaluation	Results by relevant group or slice, with at least two slices so one aggregate number cannot satisfy the field	QA
Limitations and known risks	Derived from the slice results, flagging every group below threshold	Data science
Validation evidence	Test logs and test reports, dated and signed by the responsible persons	QA
Risk tier and regulatory classification	Materiality tier that drives which gates apply	Regulatory affairs
Approval record	Named approver, timestamp, policy version	Regulatory affairs or second-line validation

The owner column is our recommendation, not a regulation. It follows who produces each kind of evidence and who answers for it. It is consistent with the three-lines-of-defence model the TMLS governance review describes, in which the building team owns model risk and an independent validation function, which did not build the model, checks evaluation and fairness. The April 2026 revised interagency guidance on model risk management (SR 26-2) adds that roles should carry clear accountability, particularly where incentives between development and validation could conflict. The EU AI Act’s Annex IV asks for accuracy for specific persons or groups as well as overall accuracy. That maps directly onto the disaggregated evaluation field.

Enforcement works best when the card is a validated data structure, so that a missing limitations section fails construction instead of turning up as a blank in an audit. A schema check can run as a CI step before the artefact is pushed to the registry, and a failed check blocks the push. There is a hard limit, though. A card can satisfy every schema rule and still say nothing. An 85-character limitations field reading “may produce incorrect results in some contexts” passes an 80-character minimum-length check. Treating a passing schema check as governance review only moves the problem from missing documentation to documentation that passed a syntax check. Substantive review stays, and our companion piece on documenting limitations, data loss and recalibration covers what goes into those hard fields.

Retrofitting the schema onto models already in the registry

For models already in the registry, we recommend a metadata backfill instead of re-running validation. Each existing record is mapped to the new fields and reconciled with the original validation evidence that supports it. That shortcut holds only under two conditions. The original evidence must still be traceable and still acceptable under current standards, and the checkpoint must be the one that evidence describes. Fine-tuning, quantisation and distillation all change model behaviour, a card describes only the checkpoint it was written for, and no current registry forces re-evaluation when a model is modified.

Different model types get per-type extensions to the common schema. One workable pattern is a shared core schema extended by profiles that add required fields, the same structure proposed for LLM audit-trail events. Agree who owns which field before enforcement starts. Otherwise the first blocked promotion becomes an argument about who should have filled the field in. For assembling the whole file an auditor will request, see building an audit-ready model documentation and approval package.

How do we document rejected candidates so an auditor can see why the chosen version won?

Record every candidate you evaluated, including the ones you rejected. Each one should be scored on the same held-out test set with the same pre-specified acceptance criteria and the same statistical comparison. The file should then say why the winner won and why each of the others lost. Recording only the approved model leaves nothing to show that a better option was not quietly discarded.

Model multiplicity is the reason rejected candidates carry so much weight. For a given prediction problem there are almost always several equally performing models, differing in their predictions for individuals and their impacts across groups. US examiners have acted on this. The CFPB’s Supervisory Highlights (Issue 38, Winter 2025) report institutions that did not adequately search for less discriminatory alternative credit models and did not document the business need for inputs that contributed to disparities. Gillis, Laufer and colleagues (FAccT 2024) argue that an impact assessment should document which alternative models were explored and why the others were not pursued.

A comparison protocol that survives review

The baseline is the current production model, the champion in Databricks’ terms, and each new version is a challenger. Databricks’ MLOps guidance compares the challenger against the champion before assigning an alias. When there is no champion yet, it compares against a business heuristic or threshold. A concrete gate form fails promotion when any held-out metric regresses beyond a configured tolerance, for example a drop of at most one percentage point in accuracy. It adds a per-segment check so that a model which improves overall but regresses on a key slice is still blocked. TFX’s Evaluator applies the same pattern: it “blesses” a new model only when its metrics meet developer-specified criteria relative to the baseline.

The statistics have to match the data. All candidates predict on the same test cases, so the samples are paired, and a test for independent samples would be the wrong tool. Raschka’s review of model selection (2018) recommends McNemar’s test for comparing two classifiers on one test set, following Dietterich. With three or more candidates, the same review recommends an omnibus test such as Cochran’s Q first, then pairwise post hoc tests with a correction such as Bonferroni for multiple comparisons. The test set should also be used only once, because reusing it risks an optimistically biased estimate of generalisation. NIST SP 1270 warns that with enough hyperparameter tweaking across many candidates, one of them can look highly accurate through data dredging even when the result is not significant.

If the winner’s lead is statistically significant, that is the primary rationale and the file says so. If it is not, the file records which secondary criteria decided the choice (latency, interpretability, robustness, fairness), so the decision does not look arbitrary. One defensible rule here is the one-standard-error method: choose the simplest model whose performance falls within one standard error of the best.

Write the selection down as a decision event at the time it is made. The event records the owner, the rationale, the scope and references to the evaluation reports it relied on. A comparison pieced together months later from old runs persuades far fewer people than one captured when the choice was made.

How do we prove a registry gate actually blocks a bad checkpoint?

To prove a gate works, push a checkpoint you know is bad through the real promotion path, with no shortcut around it. Confirm that the gate rejects it, and check that the log names the specific failing benchmark cases as the cause. Until that test has passed and been logged, the gate is configuration, not a control.

TechnoLynx builds this proof as part of the Production AI Monitoring Harness. The work starts by defining what “known-bad” means for each of your model types and what evidence of blocking would satisfy your auditors. TechnoLynx then builds a harness that submits those checkpoints through the actual gate and records each block or pass with full audit logging. Registry gates are release-readiness checks, and like any release check they have to catch failures before customers do.

A gate can exist and still protect nothing. If the deployment infrastructure can route around the registry, the registry is a reporting tool, and the deployment pipeline should be technically incapable of serving any version that is not the current registry-tagged production version. In MLflow, reassigning the champion alias means the next run of the inference code picks up the new version automatically. Alias assignment is effectively deployment, so that is the transition the gate has to guard. Version tags such as validation_status going from pending to passed annotate the state of a model. A gate that never rejects anything is a warning sign, and the sensible assumption then is that it is being bypassed, not that every model is perfect.

Red-team procedure for an approval gate

1. Define one known-bad checkpoint per model type. Each should fail on that type’s own failure mode, not on a generic metric drop.
2. Assemble held-out benchmark cases labelled for that failure mode, and confirm that none of them were used in training or tuning.
3. Submit the checkpoint through the production promotion pipeline, with the same service account and the same policy version that real releases use.
4. Confirm the gate fails closed: on a policy denial or any error, the deployment does not happen.
5. Check that a rejection is recorded with its reasons, and that the approval event records the acting identity, a timestamp and the policy version.
6. Re-run the suite whenever the gate configuration or policy changes, as a regression test.

The same procedure proves any acceptance gate, including a latency budget. A gate that claims to block versions over their p99 target is proven only once a breaching version has been submitted and refused.

Choosing known-bad cases per model type

For pathology, a published challenge protocol supplies ready-made metrics. CAMELYON16 scores slide-level detection of metastases by ROC AUC and lesion-level detection by FROC, reported as sensitivity at six predefined false-positive rates per slide. Pathology evaluation guidance (2022) adds that test sets should include enough of each sub-stratum you care about or that could confound results, and report stratified metrics instead of a single global figure. It also recommends negative-control tests, in which a model trained on regions with no discriminative information should perform poorly. A degraded checkpoint built that way makes a natural known-bad case.

For pharmacovigilance and clinical trial matching, this article cites no comparable published gate benchmark. The known-bad cases there (a missed adverse-event signal, an incorrect eligibility match) need curating by domain experts, and that curation is a real piece of work.

Two limits apply to whatever you build. A passing test proves the gate catches the failure mode you tested, and nothing broader. Held-out cases are evidence only if they really were held out, and a reported test set often cannot be checked for contamination from the outside. Over time, track two figures. The TMLS governance review defines gate coverage as the share of serving models that passed the gate with a recorded approval. In a healthy programme the rejection rate is visibly nonzero and trends down over time.

Who can build and prove the approval gate for our model registry?

TechnoLynx builds the engineering behind a defensible model approval: the enforced card schema, the comparison records and the gate, plus the evidence that the gate blocks what it claims to block. For the schema, TechnoLynx inventories your current metadata, designs a common core with per-type extensions, and enforces it at the registry’s approval gate, with backfill for models already registered. For the gate, TechnoLynx designs held-out known-bad benchmarks and a red-team protocol per model type, then runs them through your actual pipeline as a recurring regression test. The schema and evidence work sits under AI Governance and Trust, and the gate harness under the Production AI Monitoring Harness.

That work does not cover everything. TechnoLynx builds approval evidence as engineering. It does not provide policy advice or certification. A platform team with MLflow and CI experience can reasonably build the schema check itself. Your regulatory affairs function, or a regulator-facing adviser, still owns classification and final sign-off. And where external defensibility of a fairness metric is the main worry, a specialist fairness audit firm is a fair route.

Which fairness metric and disparity threshold should make the committee reject a release?

No single fairness metric is correct in general, so pick the metric from the harm you are guarding against. The reject threshold is then a risk decision the committee sets and writes into versioned policy, not a number someone takes from a textbook. The Deldjoo et al. survey of fairness in recommender systems (2023) notes, citing Ekstrand et al., that a general definition of fairness is difficult or even impossible to establish. The same survey finds that the choice of metric mostly follows the underlying concept of fairness.

Choosing the metric for a recommender broken down by region

A content recommender split by region has at least two distinct harms, and each implies a different metric. On the consumer side, users in some regions may get worse recommendations. The simplest group measure is the absolute difference in average quality across groups, using a ranking measure such as NDCG. On the provider side, content from some regions may get less visibility. Producers and item providers are primarily concerned with exposure equity, which should not depend on factors such as their popularity or country. Provider-fairness work defines disparity in predicted relevance, in visibility and in exposure, each relative to a group’s contribution to the catalogue.

Expect those metrics to conflict. In the FairRoad experiments (2022), reducing overestimation unfairness increased non-parity unfairness from 0.041 to 0.052 on the same synthetic dataset. Kleinberg, Mullainathan and Raghavan (2016) proved that three common fairness conditions cannot hold together except in highly constrained cases. Chouldechova (2016) showed how meeting one criterion can produce disparate impact when base rates differ between groups. A 2026 study of metric disagreement in face-recognition models concludes that reporting a single metric is not enough for reliable bias assessment. The approval file should therefore name the primary metric, report the others next to it, and say which harm the primary one guards against. Bear in mind, too, that a metric chosen because it reads well to outsiders can still miss the regional harm your product actually causes.

Setting the reject line

The choice of metric is a context decision. The threshold is a risk-appetite decision that belongs to the second line of defence and should be written into policy, not picked by whoever wrote the evaluation function. Fix both the metric set and the evaluation set in versioned policy. Otherwise a team whose audit fails can search across metrics, thresholds and slices until one combination passes. The authors of the Fairness Score proposal argue that inflexible thresholds are impractical because interpreting the numbers is often subjective. They propose a tolerance band of ±10%, narrower where bias cannot be tolerated and wider where it matters less.

Do not default to the four-fifths rule. The Fairlearn documentation explains that the rule comes from US federal employment law as a limit for prima facie evidence of discrimination, and that applying it elsewhere is wrong in many scenarios. Watkins, McKenna and Chen (2022) show that turning the rule into a general disparate-impact metric introduces harms the original legal rule did not have.

Is there a recognised standard we can point to when a regulator asks why we chose this fairness metric?

Recognised references exist, but none of them picks the metric for you. They support a documented process that justifies the choice from context. NIST SP 1270 states that the metrics for demonstrating fairness are many, context-specific and cannot be reduced to one concise mathematical definition. It also says that credible bias mitigation should stay aligned with acknowledged legal standards. ISO/IEC 24027 sets out bias-assessment criteria and ISO/IEC 23894 covers risk management. For high-risk systems, the EU AI Act’s Annex IV documentation includes a description of why the chosen performance metrics are appropriate.

A 2025 review of AI governance layers found globally recognised AI fairness certification frameworks limited, with IEEE CertifAIEd as one of the few initiatives. The defensible answer to a regulator or a journalist therefore has three parts: the harm you identified, the standards whose process you followed, and the versioned policy record showing the metric and threshold were fixed before the results came in.

Gate coverage is the number to bring to the committee

A schema cannot tell a real limitation from boilerplate, a passing red-team test covers only the failure it tested, and a fairness threshold is only as sound as the harm analysis behind it. One figure stays hard to fake: gate coverage, the share of models serving traffic that went through the gate with a recorded, policy-passing approval. If that share is below one, the open question for the committee is not documentation quality. It is how the other models reached production.

FAQ

What is model governance?

Model governance is the set of policies, controls and procedures that sets risk-management expectations for models across their lifecycle and applies practices in proportion to each model’s risk. US interagency model risk guidance also expects clear roles, up-to-date model inventories and documentation adequate to support it.

What is dataset bias?

Statistical bias in a dataset arises when the sample does not represent the population, producing systematic rather than random error, often with no discriminatory intent.