A perception validation evidence pack is not free-form prose. It is a structured document in which every rendered figure must reconcile to the audit run that produced it. Treat generation as loose text templating — stitch metrics into a narrative and hope the numbers stay consistent across sections — and the pack drifts silently the moment the audit schema changes. A reviewer catches the mismatch at the gate, the pack bounces, and you lose a round-trip you did not budget for. A structured generation framework like SLIME approaches the same problem from the opposite direction. Instead of laying text over data and hoping the binding holds, it binds each rendered element to its underlying audit data model, so the pack is produced as a typed, layout-managed artifact rather than assembled prose. The consequence is not cosmetic. When a field no longer maps to source, generation fails loudly — at build time, on your machine — instead of shipping a plausible-looking document whose mAP figure in the summary no longer matches the mAP figure in the detail table. That difference is the whole argument of this article. If you are assembling release evidence for an automotive perception model, the mechanism you choose to turn audit data into a document decides whether the pack is traceable by construction or hand-reconciled under time pressure. This is document generation tooling, to be clear — not safety certification. But it is the tooling that determines whether the certification conversation starts from evidence you can defend or evidence you have to re-verify by eye. What does the SLIME framework do, and what does it mean in practice? Names in this space get overloaded, so it is worth being precise about the class of tool rather than the acronym. A structured generation framework, in the sense that matters here, is one that treats a document as a tree of typed nodes bound to a data model — not as a string with holes punched in it for interpolation. Each node in the pack (a metric cell, a confusion-matrix figure, a per-class recall row, a dataset-version footnote) declares what field of the audit run it renders. The framework resolves those bindings when it generates the document, and a binding that cannot resolve is an error, not a blank. In practice this means the audit run — the actual output of your evaluation harness, with its per-class scores, its slice breakdowns, its dataset and model version stamps — is the single source. The pack is a projection of that source. You do not copy a number into a paragraph; you reference the field, and the framework renders it. When the field moves, renames, or disappears because the audit schema evolved, the framework tells you at generation time which nodes now point at nothing. Contrast that with the loose-templating path most teams start on. A Jinja or Mustache template with `` looks structured, but the template engine has no idea whether summary.map is the same quantity the detail section calls [email protected]. It will happily render both, from two paths that were supposed to be the same number, and never warn you when they diverge. That gap between “looks bound” and “is bound” is exactly where evidence packs rot. How does a structured framework bind each figure to its source audit run? The binding is the load-bearing idea, so it deserves its own mechanics. Three things have to be true for a figure to be traceable by construction: The source is addressable. The audit run is a typed data model — a schema, not a bag of JSON — so every quantity the pack can reference has a stable, declared path. The rendered node names its source. Each cell, figure, and footnote in the document declares the field it renders, rather than receiving a pre-computed value. Resolution is enforced. At generation time the framework walks the document tree, resolves every binding against the schema, and refuses to emit the pack if any binding is unresolved or type-mismatched. When all three hold, a single number appearing in the executive summary and again in a per-class detail table is not two numbers that happen to agree — it is one field, rendered twice. There is no path by which they can disagree, because there is only one source. That is what “traceable by construction” means: the traceability is a property of how the document was built, not a property a reviewer establishes afterward by cross-checking. This is the same discipline that shows up in machine learning monitoring for provenance-preserving compliance, applied one layer up. Monitoring preserves the provenance of the audit run; structured generation preserves the provenance of the document that reports it. Break either and the chain from model behaviour to shipped evidence has a gap. How structured generation differs from loose text templating The two approaches look similar in a demo and diverge under maintenance. The distinction is not “template vs. no template” — a structured framework has templates too. The distinction is where the type information lives and when binding failures surface. Dimension Loose text templating Structured generation framework Unit of assembly String with interpolation slots Typed document tree bound to a data model Source of a figure Whatever value is passed in scope Declared field path into the audit run Same number, two sections Two independent interpolations, can silently diverge One field rendered twice, cannot diverge Schema change Renders stale or blank; no signal Unresolved binding raises a generation-time error Failure timing At the reviewer gate, after hand-off At build time, before hand-off Layout Manual; drift needs manual fixes Managed by the framework Traceability Established by review A property of construction The row that matters most for release work is the failure-timing one. A loosely templated pack fails at the gate, which is the most expensive place to fail — it consumes a reviewer’s attention, burns a round-trip, and the defect might not be the mismatch itself but the loss of trust in every other figure once one is caught wrong. A structured framework moves that failure earlier and cheaper. This is a close cousin of the reasoning-strategy choice we cover in chain-of-thought vs tree-of-thought for evidence-pack assembly: the question is always where you want the system to be forced to be explicit, and the answer for release evidence is “as early as possible.” What happens at generation time when the audit schema changes? This is where the abstract “fails loud, not silent” claim earns its keep. Audit schemas are not static. A perception team adds a new object class, splits a metric by weather condition, renames a slice, versions the dataset. Each of those is a schema change, and each is routine. Under loose templating, a renamed field is invisible to the template engine. If results.night.recall becomes results.slices.night.recall, the old interpolation resolves to nothing or to a stale cached value, and the pack renders anyway. The failure is deferred to whoever reads the document — best case a reviewer, worst case nobody until an auditor asks where the night-recall figure came from. Under a structured framework, the document node that declared a binding to results.night.recall now points at a field that does not exist. Resolution fails. Generation halts with a specific, locatable error: this node, this expected path, this schema. The engineer who changed the schema learns immediately that a pack referenced the old shape, and fixes the binding as part of the same change that moved the field. The schema and the document that reports it stay in lockstep because divergence is a build break. The honest caveat: the framework guarantees the binding is valid, not that the number is correct. It ensures the figure in the pack is the figure the audit run produced. Whether the audit run itself measured the right thing is a separate question — one that the performance metrics discussion is really about. Structured generation removes the transcription-error class of defect entirely; it does not remove the measurement-design class. Which parts of a pack should be generated, and which stay human-authored? A structured framework is a scalpel, not a floodlight. Pointing it at the whole document is a mistake — some of a validation pack is genuinely narrative, and forcing narrative through a data-binding framework produces stilted, over-constrained prose that reads worse and helps no one. The clean split follows a single test: does this element reconcile to a number in the audit run? If yes, it should be generated and bound. If it is judgment, context, or argument, it should be human-authored and reviewed as prose. Generate and bind: metric tables, per-class and per-slice score breakdowns, confusion matrices, dataset and model version stamps, coverage summaries, threshold-pass/fail indicators, run identifiers and timestamps. Everything whose ground truth is the audit run. Human-author: the rationale for why a given metric threshold is appropriate for this operational design domain, the interpretation of a borderline result, the residual-risk argument, the scoping statements about what the pack does and does not cover. The framework’s value is proportional to how much of the pack is figures that must reconcile. For a perception validation pack — dense with per-class detection metrics across slices, across dataset versions — that fraction is high, which is exactly why the tooling pays off here more than it would for a mostly-narrative document. In measurable terms, what does structured generation improve? The ROI shows up in three places, and all three are observable without a controlled study — you can read them off your own gate logs before and after adopting the approach. First-pass reconciliation rate — the share of regenerated packs that pass a reviewer’s figure-to-source reconciliation on first hand-off. Binding-by-construction removes the transcription-mismatch class, so the reviewer stops finding “this number doesn’t match that number” defects (observed pattern across evidence-automation engagements; not a benchmarked rate, and the size of the gain depends on how mismatch-prone the prior process was). Layout rework hours per gate — time spent manually re-fitting figures, tables, and page breaks when a pack is regenerated. A framework that manages layout as part of generation collapses this toward zero; the hours saved are directly countable from time tracking. Schema-drift defects escaping into a shipped pack — the count of figures that reached a shipped document while pointing at a stale or renamed field. Under a structured framework this count is bounded at zero by construction, because such a binding cannot generate. Note the evidence classes deliberately: the first is an observed pattern whose magnitude is engagement-specific, the third is a structural guarantee rather than an empirical rate. Do not let anyone quote “eliminates reviewer round-trips” as a benchmark — it is a mechanism claim about which class of defect is removed, not a measured percentage that will hold in your environment unchanged. Where does a structured generation framework stop? Here is the boundary, stated plainly so no one over-reads the tooling. A structured generation framework produces a traceable document. It does not constitute safety certification, and it makes no claim about the adequacy of the evidence the pack contains. It guarantees that every figure reconciles to its source audit run. It does not guarantee that the audit run tested the right scenarios, that the coverage is sufficient for the ASIL target, that the metric thresholds are defensible, or that the model is safe to ship. Those are engineering and assessment judgments that live outside the document-generation layer. A perfectly traceable pack can still describe an inadequate validation campaign — the framework will render that inadequacy faithfully and traceably. That is the correct division of labour. The framework’s job is to make the document an honest, drift-proof projection of the audit run, so that the human review can spend its attention on whether the evidence is sufficient rather than on whether the numbers were transcribed correctly. Getting the second question off the reviewer’s plate is precisely what lets them do the first question well. The same structured-generation discipline generalises beyond perception — it applies to reliability-audit output and other release-evidence documents wherever figures must reconcile to a source run, a connection worth following if your evidence work spans more than automotive. For the perception case specifically, this framework is the mechanism underneath a computer vision validation practice that treats the evidence pack as Production AI Monitoring Harness — generated and traceable by construction, not hand-collated and re-verified by eye. FAQ How does the SLIME framework work? It treats a validation pack as a tree of typed nodes bound to a data model rather than a string with interpolation slots. In practice, the audit run is the single source, each rendered element declares the field it reports, and the framework resolves those bindings when it generates the document — so the pack is a projection of the audit data, not a hand-copied narrative. How does a structured generation framework bind each rendered figure in a validation pack to its source audit run? Three conditions make it work: the audit run is an addressable typed schema, each rendered node names the field it renders instead of receiving a pre-computed value, and the framework enforces resolution at generation time. A number appearing in two sections is then one field rendered twice — it cannot diverge, because there is only one source. How does structured document generation differ from loose text templating or narrative report generation? Loose templating lays text over data and hopes the binding holds; two interpolations that should be the same number can silently disagree, and failures surface at the reviewer gate. A structured framework keeps type information in the document tree, so an unresolved or mismatched binding raises a build-time error before hand-off. Both use templates — the difference is where types live and when failure surfaces. What happens at generation time when the audit data schema changes? A renamed or removed field means a document node’s declared binding no longer resolves, so generation halts with a specific error naming the node, the expected path, and the schema. The engineer who changed the schema fixes the binding in the same change, keeping the document and the audit run in lockstep. It guarantees the binding is valid, not that the underlying number was measured correctly. Which parts of an evidence pack benefit from a structured framework, and which should remain human-authored? Anything that reconciles to a figure in the audit run — metric tables, per-class and per-slice breakdowns, version stamps, pass/fail indicators — should be generated and bound. Judgment, interpretation, threshold rationale, and residual-risk argument should stay human-authored. The framework’s value scales with how much of the pack is figures that must reconcile. In measurable terms, what does a structured generation framework improve? Three observable places: first-pass reviewer reconciliation rate (transcription-mismatch defects are removed by construction), layout rework hours per gate (managed layout collapses manual re-fitting), and schema-drift defects escaping into a shipped pack (bounded at zero, since a stale binding cannot generate). The first is an engagement-specific observed pattern; the third is a structural guarantee, not a benchmarked rate. Where does a structured generation framework stop, given it produces traceable documents but does not constitute safety certification? It guarantees every figure reconciles to its source audit run and nothing more. It does not judge whether the audit tested the right scenarios, whether coverage meets the ASIL target, or whether the model is safe to ship. A perfectly traceable pack can still describe an inadequate validation campaign — the framework renders that faithfully, freeing reviewers to judge sufficiency rather than check transcription. If you are choosing how to turn audit data into release evidence, the question to sharpen is not “which framework has the nicest output” but “at which point in the lifecycle does a figure that no longer maps to source get to fail” — because that single choice decides whether your pack is traceable by construction or reconciled by hand at the gate.
