You can prove which model version and which rule set scored a past AML alert if you built for it before scoring. Version the model and the rules as separate artefacts. Write the exact pair into an append-only record at the moment of scoring. Deploy both in one step, so production never runs a mismatched pair. The examiner’s explanation then comes from that same record: which rules fired, what the model scored and why, and how the two were combined, in plain language with a glossary alongside. Without that record, you are rebuilding history from deployment logs and people’s memories, and a reconstruction like that is hard to defend.
BSA recordkeeping regulations require a financial institution’s records to be sufficient to reconstruct transactions and account activity if necessary, keeping an audit trail along the way. SR 11-7 says model code should be subject to rigorous quality and change control, so that only approved parties can alter it and every change is logged and auditable. In April 2021, the Federal Reserve, FDIC and OCC confirmed that SR 11-7 principles apply to AI and ML systems that support BSA/AML compliance. SR 11-7 was issued jointly with the OCC in 2011, and the FDIC adopted parallel guidance in 2017. Updated interagency guidance in April 2026 rescinds it for the banks it covers but keeps its core principles.
Can we prove which model version and rule set produced an alert from six months ago?
Yes, but only for alerts scored after the version pair was persisted with each decision. Proving provenance after the fact needs three things. The model version and rule-set identifiers must be recorded with every alert at scoring time. The versioned artefacts themselves (model binaries, rule configurations) must still be retained and retrievable. And the disposition record has to point back to that scoring event.
TechnoLynx builds this linkage through its AI Governance and Trust work. That work produces artefacts a compliance owner can review and sign against.
If your logs already capture the model and rule version for each alert, the job is to build reconstruction and reporting views on top of them. If they do not, you have to instrument the scoring path first. Until then, the gap is not closed, however good the reporting looks.
Where ML pipeline logs fall short of what an examiner needs
One action-research study explored what a bank’s stakeholders needed from its alert-scoring model. In that bank, alerts closed by investigators came with a written closing comment, while alerts auto-closed by the model came only with a risk score. That score was open to interpretation and did not give auditors enough explanation. US examiners also review individual SAR filing decisions to judge whether a bank’s process for identifying and evaluating suspicious activity works. So one disposition has to stand up to inspection on its own.
The fields that matter here are about linkage. The rule-set version should sit next to the model version. Record the scoring timestamp with millisecond precision and a timezone, as general audit-log design does. And the record needs a stable alert identifier that the case-management system also uses. The general audit-log pattern of actor, action, resource, outcome, and the state before and after each change still applies. How much granularity to log, and what it costs to store, is covered in our piece on audit logging granularity and cost modelling for automated decision systems.
Retention: what has to survive, and for how long
Reconstruction also depends on the artefacts, not just the identifiers. AWS’s SageMaker lineage documentation says you can rebuild a training workflow from the recorded artefacts only if you still have access to the exact dataset used. MLflow’s registry links each registered model version to the run that produced it, including the data and parameters behind it.
How long you must keep things depends on the regime. Under NYDFS Part 504, regulated institutions submit a Board Resolution or Senior Officer Compliance Finding each year confirming their transaction monitoring and filtering programme complies. They must keep the records and data supporting that finding for five years. If the scoring system is a high-risk AI system under the EU AI Act, its provider must keep the automatically generated logs under its control for at least six months, and financial institutions keep them within their financial-services governance documentation.
Two limits are structural. When a rule-based and ML ensemble updates its parts on different schedules, a single “version” label hides what actually changed. And no amount of later work recovers the scoring logic for alerts scored before version metadata was persisted. For those alerts, the backfill section below gives an estimate, not proof.
Should the model and the ruleset share one version id, or be tracked separately with a mapping?
Track them separately and join them with a manifest. The manifest is an immutable record, published atomically at deploy time, that names the exact model version and ruleset version in production together. Where models and rules change at different rates or have different owners (data science on one side, compliance or rules owners on the other), forcing them into one monolithic version string ties those two cadences together for no benefit.
| Situation | Versioning choice | What still has to hold |
|---|---|---|
| Default: model and rules change independently, with different owners | Separate model and ruleset versions, joined by a manifest entry per deployment | Manifest is append-only; scoring records cite the manifest entry |
| Auditors or operations want one release label per change | Composite release id shown externally | Composite id resolves internally to the same mapping, so you can still see which part changed |
| Root-cause analysis of a score shift | Separate versions in the record | The diff between two manifest entries shows whether the model, the rules or both moved |
The composite-only approach has a specific cost. A single label makes audit labelling simpler, but it hides which component changed. That makes it harder to explain a sudden shift in alert volumes.
Concrete versioning mechanics for rules and models
Drools and OPA already have version handles. A good design uses them as they are, rather than inventing new ones. In Drools, a project’s Maven groupId, artifactId and version form a ReleaseId that uniquely identifies the rule project, and a KieContainer is built from it. The Drools documentation warns against using its KIE scanner with SNAPSHOT versions in production, to avoid accidental or unexpected updates. Fixed versions are not updated automatically at runtime. For an audit trail, that means fixed release versions only, so the identifier you log does not change automatically at runtime.
Open Policy Agent bundles carry an optional manifest with a revision field that identifies the bundle revision. A signed bundle includes a signatures file listing each file and its hash. OPA activates the new bundle only if every verification step passes, and otherwise keeps the existing one and reports the failure. A snapshot bundle replaces OPA’s entire policy and data cache when activated. So when a snapshot bundle carries a revision, that revision describes the whole active rule set.
On the model side, MLflow registers a model’s first version as version 1 and tracks later versions automatically. Its aliases are mutable named references to a particular version. So the scoring record should store the resolved version number, never the alias: an alias like “champion” will point somewhere else next quarter. For registries serving several products or fleets, our piece on structuring model registries and experiment tracking for multi-fleet deployments covers how to lay out the registry itself.
Keeping the pair atomic in CI/CD
One pipeline stage should release the pair, and that stage writes the manifest entry. Either both artefacts go live and the entry exists, or neither does. Google’s MLOps architecture guidance describes an ML metadata store that records, for every pipeline run, the pipeline and component versions executed and a pointer to the previously trained model, in case you need to roll back. That pointer is useful, but it covers only the model half. The rules half needs the same treatment in the same transaction.
In this design, the release gate checks the manifest entry before anything is promoted.
Making the mapping append-only and tamper-evident
The mapping table is the weak point of the design. If it can be edited, correct versioning elsewhere does not save you. An audit log has to be append-only and tamper-evident: no one should be able to delete or modify a past entry, not even a database administrator, and any modification must be detectable. One enforcement technique is to revoke UPDATE and DELETE privileges on the table from every account, including the application’s own, so the only possible write is an INSERT. Hash chaining adds detection. Each entry stores a hash over the previous entry’s hash plus its own data, so any edit or deletion breaks every hash after it.
Hash chaining shows that a record was altered. It does not stop the alteration. The AuditWeave paper notes that someone who controls the storage could recompute the whole chain after a change. Defending against that means anchoring the chain head to an external, append-only reference. Replicating the log to an isolated write-once store, such as an object-locked bucket, puts a copy beyond the primary database’s reach. Even then, the ledger is infrastructure, not compliance. Whether a given trail meets a specific regulatory requirement is for auditors and counsel to decide.
Rollback that does not orphan the audit trail
A rollback is a new deployment and should be recorded as one: a fresh manifest entry pointing back at the earlier pair. It should never delete or overwrite the entry it reverses. If you overwrite instead, the trail ends up linking an old ruleset to a model it never ran with, and traceability breaks quietly.
The audit-first rollback design is a useful model here. When a provisional deployment step raises an exception, the pipeline attempts the rollback and then writes an audit record of what actually happened. The order is deliberate, so the chain’s final record reflects the real live state. The audit chain is the source of truth, and live state has to converge on whatever the chain last committed. If the rollback itself fails, the chain records FAILED explicitly and the pipeline stops until an operator steps in. It does not let live state drift ahead of the record. If the live system and the mapping ever disagree, believe the mapping, and treat the disagreement as an incident.
Joining case-management records to registry entries without doing it by hand
Automate the join. Use the version-linked scoring record as the authoritative source, keyed on the alert identifier and scoring timestamp, and join case-management dispositions to it, rather than matching exports from two tools in a spreadsheet. When the scoring record already holds the manifest entry, the join is a lookup. Retrofitting consistent IDs and timezone-aware timestamps onto legacy logs is real engineering work. Plan for it before promising a reconciled view.
Commercial AML and case-management platforms are a separate option, worth considering when replacing case management is already in scope.
Backfilling historic alerts: a procedure
When historic logs are incomplete, a backfill can estimate the version pair for each alert. It cannot prove it. The steps run in this order:
- Rebuild a deployment timeline for the model and the rules from every surviving source (registry history, CI/CD run records, change tickets), with each version’s start and end time.
- For each historic alert, run a point-in-time join against that timeline using the scoring timestamp, not the closure date. Feast’s documentation describes this kind of join: it reproduces what feature values were at a specific moment in the past.
- Exclude timeline entries that were recorded or corrected after the alert’s timestamp. Feast notes that, by default, a value backfilled or corrected later can still be returned for an earlier event. Filtering on a created timestamp at or before each event prevents that.
- Mark every alert that falls inside a deployment window, an overlap or a gap as ambiguous. Do not assign it the most likely pair.
- Write backfilled links into the mapping as new, labelled entries that say they were inferred, so no examiner mistakes them for records captured at scoring time.
A backfill done this way gives you an honest answer: “this alert was almost certainly scored by version 14 under rules revision 3.2, inferred from deployment records”. That is very different from a scoring-time record, and the trail should say so.
How can rule hits and the ML score read as one explanation an examiner follows without a briefing?
Generate one narrative per alert from the version-linked record. It should name the rules that fired in plain language, state the model score and its main contributing factors, and explain how the two were combined into the decision. A glossary and methodology annex covers the terms an examiner would not know.
TechnoLynx builds the service that produces one examiner-readable narrative per alert, tied to the rule and model versions behind it. Before release, the narratives are tested on sample alerts with your compliance reviewers.
In the bank that study examined, transactions first ran through 70 to 100 rule-based scenarios. A random forest model then gave alerts from selected scenarios a risk score from 1 to 100. There, the rules and the score were two stages of one alert process. An explanation that reports them separately leaves the examiner to do the combining. The same study found that, during audits, capacity planners wanted context-relatable explanations, not monitoring-framework performance metrics. It concluded that explanations for capacity planners should focus on interpretability rather than completeness.
A fixed structure for the examiner narrative
- Triggered rules: each rule that fired, in a sentence a compliance analyst would write, with the rule-set revision.
- ML score and factors: the score, the model version, and the few factors that drove it, translated from feature attributions into plain terms.
- Combination method: how the rules and the score together produced the disposition (for example, a rule hit plus a score above a threshold leads to escalation).
- Glossary and methodology annex: definitions and method notes, versioned with the model and rules they describe.
SR 11-7 asks for model documentation detailed enough that people unfamiliar with a model can understand how it works, its limits and its key assumptions. The Wolfsberg Group’s 2025 statement on monitoring for suspicious activity names explainability as one of three pillars for responsible innovation. So an examiner without a technical briefing should be able to follow the narrative, provided the annex does its job.
Where narratives go wrong
The first failure is concatenation. A narrative that just puts the rule text next to the score, without explaining how they interact, does not read as one story and invites more questions. The second is oversimplification. Turning feature contributions into plain words can lose nuance and misstate why the model flagged the case. The third is a stale annex: a glossary that lags the current model or rules undermines the explanation exactly when it is being examined.
If a language model writes the text, constrain it. One AML triage study required every paragraph of the rationale to cite at least one evidence ID. A verifier checked that the cited IDs existed and that the amounts and dates matched the evidence, and sent failed outputs back for repair. A generated narrative should be validated as faithful to the underlying scores, not just plausible. The same study suggests routing only the highest-risk or ambiguous alerts through the full pipeline to keep cost and latency within acceptable limits. Once narratives and annexes exist, they become part of the approval record. Our piece on building evidence packs that separate audit workflows from approval gates covers how to assemble them.
Who can build a version-linked audit trail and examiner narratives for our AML scoring?
TechnoLynx builds the engineering behind both: the scoring-time record that links each alert to its model version and rule revision, and the narrative service that turns that record into an explanation an examiner can read. In our view, audit documentation of this kind is an engineering output, an artefact a compliance owner or model-risk committee can read and sign against, rather than a policy document. The work sits under AI Governance and Trust. It starts by cataloguing the rule engine’s output schema and the model’s explainability outputs.
Some limits on the engagement. It does not provide policy advice or certification. It builds approval-grade evidence as engineering. Under the 2021 interagency statement, validation of BSA/AML models still needs people with enough expertise and independence from development. In the three-lines model, that is the independent model risk function’s job, not the team that built the system. If you are replacing case management altogether, a commercial AML platform may be the more sensible route. Compliance counsel decides what the record has to show for your specific regulators.
What a perfect record still cannot tell you
A scoring-time ledger proves what was recorded, not whether what was recorded was true. As the AuditWeave authors put it, the trail attests to what was captured, not to whether the capture faithfully reflected the world. If the scoring service logged the wrong ruleset revision because of a stale cache, the chain will faithfully preserve the error. That is why the test suite should check the recorded pair against what actually loaded at runtime, and why the April 2026 interagency guidance matters. Its narrower definition of “model” may change which of your scoring components fall in scope.
