An AI pentest evidence pack should let every claim trace back to four things: a named attack class, a named model checkpoint, a named evaluator and a named person who signed it off. Every result in it also has to be stated as point-in-time and per checkpoint, never as a property of “the system”. In the whitepaper for v0.5 of its AILuminate jailbreak benchmark, MLCommons defines the minimal set as seven auditable artefacts, running from a benchmark version manifest through hashed run logs to management review minutes.

TechnoLynx builds that artefact set as an engineering output under its AI Governance and Trust offer. The first step is to frame what the pack has to show for the particular customer review it will face. Getting there takes two kinds of work. The first is specialised, stage-aware adversarial testing, which we would not leave to a generic network pentest. The second is the discipline of mapping, scoring and packaging what that testing finds.

The seven artefacts a reviewer will try to re-trace

MLCommons lists the minimal artefact set as follows.

#	Artefact	What it contains
1	Benchmark version manifest	Seeds, evaluator configuration, scoring, grade bands
2	System information	CPU model and core count, GPU type, memory, driver version, library and framework versions
3	Run log and hash registry	Prompts, responses, timestamps, model and version, attack configuration
4	Per-hazard report and Resilience Gap table	Baseline versus jailbreak results per hazard
5	Risk assessment update	AISA excerpt referencing the above
6	Statement of Applicability entry	Selected treatments and their justification
7	Management review minutes	Trends and decisions

The same whitepaper says the evaluator’s configuration, calibration evidence and any human sampling plans should be held as controlled, documented information, with the people who operate the evaluator named under competence and roles. Its list of controlled items covers data and prompt manifests with hashes, evaluator and scoring versions, configurations, run logs, reports, incidents and corrective-action records. Its control methods include versioning and approval before use.

MLCommons defines the Resilience Gap as the baseline non-violating rate minus the non-violating rate under attack. It is reported per hazard and overall, and the same evaluators are used for both so the two figures can be compared directly. In the v0.5 results, none of the 39 text-to-text systems tested scored better for jailbreak resistance than for safety, and 35 of them received a lower grade. We would make the Resilience Gap the centre of any language-model section of a pack. A reviewer can read a delta against a baseline without having to trust an absolute score on its own.

Evaluator uncertainty belongs in the pack too. The MLCommons v0.5 evaluator has a 17.1% false non-violating rate and a 29.7% false violating rate, an accuracy the authors call lower than they would accept for a v1.0 security benchmark.

Labelling every test with one attack vocabulary

NIST’s adversarial machine learning taxonomy (NIST AI 100-2e2025, March 2025) was written to give a common language for assessing and managing the security of AI systems. It classifies attacks along five dimensions: AI system type, the life-cycle stage at which the attack is mounted, the attacker’s goals and objectives, their capabilities and access, and their knowledge of the learning process. Those five dimensions work well as the label set for each line in the run log.

We would weigh the access dimension most when deciding what a pack can claim. In NIST’s white-box threat model the attacker has full knowledge of the model’s architecture and parameters. In the black-box model the attacker can only query the trained model and observe its predictions. A face-spoof at a turnstile is a deployment-time attack on integrity, while abliteration relies on white-box modification of the weights. When one pack holds both, the reviewer needs to see which claims rest on full access and which rest on queries alone.

Can anyone prove our face-recognition gate can’t be fooled by a printed photo, a mask or a sticker?

A failed attack cannot prove that a face-recognition gate cannot be fooled. What we would expect an independent, specialised red team to give you before go-live is a written test plan and a report. That report should say what a realistic attacker achieved against your cameras, which artefacts they used and what was not tried. NIST’s taxonomy records that the ID.me face recognition service found more than 80,000 attempts to fool its ID verification in the second half of 2020, using masks, deepfakes and images or videos of other people.

A study by the European Commission’s Joint Research Centre describes spoofing (called presentation attacks in current standards) as a purely biometric vulnerability that no other IT security solution shares. Testing it takes people who build physical artefacts and present them to sensors.

What the face-spoofing test plan has to cover

A credible plan treats each of these attack families as a separate line of tests:

• Presentation attacks. Printed photos, video replay and 3D masks. Replay copies the face’s motion as well as its texture, and masks defeat the depth cues that stop flat attacks.
• Physically realisable adversarial attacks. Printed eyeglass frames, stickers, hats, makeup, infrared LEDs and projected light patterns.
• Threshold and decision manipulation. BSI Technical Guideline TR-03166 lists these alongside presentation attacks as attack vectors on the biometric verification subsystem.

Researchers who studied adversarial light projection list ambient and positional lighting, and how they interact with the projected light, as factors critical to whether the attack works. In that study, impersonation succeeded in 23 and 21 of 25 attempts against FaceNet and SphereFace in a white-box setting. Against a commercial system tested black-box, it succeeded in 15 of 25. The JRC study, using low-cost self-made face models, found an average rise of around 210% between equal error rate and spoofing equal error rate, and even the commercial system tested was highly vulnerable. We take them as reason to test a turnstile. They do not predict how yours will do.

We would report with the standard metrics so a reviewer can read them. BSI defines the attack presentation classification error rate (APCER) as the share of attack presentations wrongly classified as genuine. It defines the impostor attack presentation match rate (IAPMR) as the share of impostor attacks in which the target reference is matched. Under ISO/IEC 30107-3, an evaluation of a full system reports IAPMR, while an evaluation of a PAD subsystem alone reports classification error rates such as APCER and BPCER. iBeta’s ISO 30107-3 procedure records the vendor configuration, the version and the exact capture-device configuration before any attack is applied, because the capture device affects the results.

What would a realistic attacker need, and can this be done in two weeks?

iBeta’s ISO 30107-3 procedure bounds attacker capability by cost and time. In iBeta’s procedure, Level 1 artefacts may not cost more than $30 and Level 2 artefacts no more than $300. At Level 1, a full-system test creates and applies a single subject’s artefacts within 8 hours, so 6 subjects take 48 hours. The JRC adds that its attacks were do-it-yourself, with a very limited budget. On that basis, we would scope a two-week engagement to a defined set of artefact species at a stated capability level. It will be narrower than an open-ended assessment, and the report should list which species it covered.

Can a vendor test it without us handing over the model internals?

Yes, through black-box testing, within limits. The face recognition survey literature describes black-box attacks as the case where the attacker can only query the target and observe its outputs, which is closer to real-world conditions. In the light-projection study, 7 of 10 obfuscation attempts succeeded against the commercial system in that black-box setting. Microsoft’s report on red-teaming more than 100 generative AI products makes a similar point: you do not have to compute gradients to break an AI system. A sensible middle ground is the access model BSI expects of evaluators, where comparison scores, thresholds and PAD scores are available to the tester. Black-box scoping still leaves gaps that white-box review would close, and the report should say so.

When you choose who runs the test, accredited biometric labs and adversarial-ML specialists do different jobs. iBeta’s test procedures are audited and approved under NIST’s NVLAP accreditation programme. BSI’s guideline has biometric and PAD functionality evaluated by approved testing laboratories. iBeta itself says a conformance result is not a certification of the vendor’s product. NIST’s FATE PAD report quantifies the accuracy of 82 passive, software-only PAD algorithms on 2D imagery of presentation attack instruments. Ask any candidate for previous reports and for the artefact species they actually built.

If a fake face gets through, did the detector or the matcher fail?

To find out which stage failed, we would test each stage on its own against inputs whose ground truth you know. The aim is to attribute the bypass to face detection and liveness or to watchlist matching. The survey literature describes a typical face recognition system as a camera, a pre-processing module and a recognition model; physical attacks act at the camera, digital ones on the later stages. Infrared attacks, for example, have disrupted pre-processing steps such as landmark extraction. The face detection and open-set recognition challenge literature adds that detector false accepts have to be classified as unknown by the recognition stage.

A secure biometric gate makes two decisions. Under BSI TR-03166, a positive verification needs a comparison score at or above the threshold and a PAD classification of bona fide. That gives a clean procedure for locating a bypass.

Procedure for localising a spoof bypass

1. Replay the bypass sample through detection and PAD alone, and record the PAD score and the live or spoof decision.
2. If PAD flagged the sample as a spoof but the gate still opened, the match was not gated on the PAD result, which BSI requires.
3. If PAD classified the spoof as live, that counts against the APCER, the share of attack presentations wrongly classified as genuine.
4. Feed known face crops with ground-truth identities straight to the matcher; a wrong-identity match there counts toward the false match rate.
5. Feed detector false positives and spoof crops to a reference matcher and measure how many it rejects as unknown.

The UCCS Watchlist Challenge already splits evaluation this way, into face detection and open-set recognition. It measures each part separately: detection at a fixed rate of false positive detections per image, and recognition through a correct unknown rejection rate for both unknown subjects and false positive detections. Keep a separate test suite and separate metrics per stage so that a regression in one cannot hide behind the other.

In the open-set challenge, the organisers attributed counterintuitive drops in unknown-rejection performance for two algorithms to some wrong labels in the training set. The Watchlist Challenge also found that thresholds set on validation data did not carry over to the test set for some methods, which produced large differences in performance. Use stage tests for diagnosis and system tests for the pack. BSI’s guideline does not allow a single biometric or PAD sub-component to be tested or certified on its own. The JRC notes that system-based evaluations give a very good estimate of a whole system’s real anti-spoofing capability.

Can we run an abliteration attack against this exact model checkpoint and show what breaks?

Yes, if you have the model’s weights, since abliteration is a white-box weight modification. The result is a measured before-and-after change in refusal and compliance for that checkpoint and that prompt set only. TechnoLynx can build this test as a repeatable pipeline. The work confirms the architecture and weight access, agrees the harm-category prompt set and scoring with you, implements the ablation on your hardware, runs the prompt set before and after, and hands over a harness you can re-run on the next checkpoint.

Arditi and colleagues found a single direction whose erasure from a model’s residual-stream activations stops it refusing harmful instructions, and they proposed a white-box jailbreak that disables refusal with minimal effect on other capabilities. Abliteration applies this by projecting weight matrices orthogonal to that refusal direction. It needs only a small set of contrastive prompts and no further training. The direction is usually extracted as the difference in mean activations per layer between harmful and benign prompts.

Which abliteration method applies to this architecture

Refusal directions have been shown in local, white-box settings, which a closed or API-only model does not offer. For those, the question turns into black-box jailbreak testing. Published defence work on abliteration tested Llama-3-8B and Gemma-2-9B, noting that their straightforward architecture simplified experimentation. With a novel architecture, we would first validate the method on an analogous open model before trusting a null result on the target. The Ablating Safety study lists many method families it did not evaluate, including weight-space abliteration, sparse-autoencoder steering and model merging. A pack should name the method it used and not imply coverage of the rest.

What the before-and-after comparison must report

Report refusal, attempt, utility, retention, spillover and instability as separate quantities, because a condition can lower refusals without improving task quality, or raise utility while increasing out-of-scope compliance. In that study, single-vector refusal projection raised the mean security score only from 0.46 to 0.50, while unsafe compliance went from 0.10 to 0.47. The study reports confidence intervals over prompts, seeds and decoding settings. That matters because, per the jailbreak oracle work, greedy-decoding evaluation of one attack on Llama 3.1 showed 25% success when systematic search found 39.06%.

Name the evaluator as well. JailbreakEval’s authors found that evaluators reached accuracies between 0.38 and 0.90 on the same jailbreak attempts.

The test has limits. A study of refusal geometry deliberately estimated and applied the direction on the same prompt set, which favours the attacker, and does not establish that the direction transfers to unseen prompts. The same work found that training with more diverse refusal openings made refusals harder to remove by single-vector ablation. We would therefore re-run the test after any retraining or safety patch. Deployment-layer filters sit outside the model, and the ablation number says nothing about them. Treat the transformed weights themselves as controlled material. Ablating Safety’s public release excludes transformed checkpoints, adapters and raw out-of-scope completions.

If ablation succeeds easily, is the model still adoptable?

Yes, for a defined use. That holds when compensating controls contain who can reach the weights and what an altered model could do, and when the residual risk is scored and written into the approval record, not treated as a yes-or-no verdict. TechnoLynx’s AI Governance and Trust work turns this into approval-grade evidence.

Compensating controls that can contain an ablated model

Abliteration needs the weights, so the controls that matter most protect them: access restriction, weight hashing and integrity checks, hardened inference endpoints, and logging. We would apply the same care to fine-tuning access. Qi and colleagues jailbroke GPT-3.5 Turbo’s guardrails by fine-tuning through OpenAI’s API on only 10 adversarial examples, at a cost of less than $0.20. NIST AI 800-1 suggests serving a model through an API before releasing its weights, since once weights are widely available the options to roll back are severely limited. It also recommends monitoring infrastructure with misuse classifiers, backed by human review.

In-model defences against abliteration have so far helped only partly. AMRA, a defence against abliteration, improved post-abliteration refusal scores by 2.16 points on Llama-3-8B and 14.70 points on Gemma-2-9B, the latter at a greater utility cost. Casper and colleagues put the state of the art for tamper resistance, as assessed by second-party red-teaming, at around several hundred steps of adversarial fine-tuning.

We would hold any substitute control to the test used in mature control regimes. The PCI DSS worksheet requires a compensating control to meet the intent and rigour of the original requirement, and to be validated in the specific environment where it runs. The Secure Controls Framework adds that some controls are material and cannot have compensating controls at all. If no feasible set of controls brings the risk within your threshold, reject the model for that use. For high-stakes or public-facing uses, we would start from rejection unless the controls are very strong.

Writing residual risk the approval board can act on

NIST AI 800-1 asks organisations to assess residual risk from the chosen deployment strategy and implemented safeguards. It also asks them to document why misuse risk was within the organisation’s risk tolerance at the deployment decision. For the board, express residual risk as a per-hazard delta. Show the ablated model’s violating rates against the baseline for each harm category, which also answers which categories of harmful output become reachable. Under the Secure Controls Framework, leadership evaluates the compensating control, not the control owner. The record captures the alternatives considered, the choice and its justification, and when the primary control is due. The control is then tracked, for example in a Plan of Action and Milestones.

We would treat the score as perishable. MLCommons asks for re-evaluation on model updates, domain shifts or new attack classes. NIST notes that the effectiveness of mitigations varies widely and that relying on them should rest on empirical evidence of their efficacy.

Who signs off before we tell customers the model resists abliteration and jailbreaks?

TechnoLynx recommends that only named roles who have reviewed a combined report sign off. That report should pair white-box ablation results with black-box jailbreak results, and the sign-off should go through your deployment-approval governance before any robustness claim reaches a customer. TechnoLynx’s AI Governance and Trust work builds the approval evidence pack those results go into. It is engineering evidence, not certification or legal sign-off.

MLCommons names five roles for its benchmark’s management system. A Security Lead owns the attack taxonomy and risk assessments, an Evaluator Lead owns grading and calibration, a Data Steward owns provenance, a Run Operations Lead owns logs and reproducibility, and an Audit Lead owns evidence gathering and corrective actions. Its release gates let a release proceed only after the data and evaluator pass verification, the audit checklist is complete, the Statement of Applicability is updated and the rollback plan is validated.

Reuse an existing red-team report template only if it already covers white-box ablation. Otherwise extend it before issuing the pack. In Casper and colleagues’ review of open-weight model documentation, tampering evaluations appeared for only one model, gpt-oss. They also argue that fully assessing open-weight risk requires evaluation under tampering threats, including fine-tuning, steering, editing and pruning.

Word the customer claim to match the evidence: on this checkpoint, against this prompt set and this evaluator, the Resilience Gap was this. The jailbreak oracle authors are explicit that an unsuccessful search is not a proof of safety, only a lower bound on vulnerability within a fixed budget.

We would put scattered findings into one vocabulary before showing any of this as coverage. We would re-classify last year’s ad hoc pentest findings into the new taxonomy from their original evidence, without re-running the tests. Map each to its closest category, and put anything ambiguous into an explicit, unmapped holding category for triage, not a quiet “other”. The TBM Council’s taxonomy migration guide makes the same points for cost taxonomies: tag each record with its taxonomy version, and flag large volumes in catch-all classifications for review.

We would normalise findings from several hosted models or vendors the same way. Build one crosswalk per source format into the shared labels. After that, any category with no findings and no tests across every model shows up as a coverage gap, not as evidence of safety. TechnoLynx can design and run that crosswalk across your existing finding formats and check its consistency before the coverage report is built on it. Expect some re-tagging to be ambiguous. The domain taxonomy in Slattery and colleagues’ AI Risk Repository is explicitly not mutually exclusive, so one finding can sit in several categories.

What does a taxonomy-coverage evidence pack look like for a renewal review?

For each taxonomy category, a coverage pack should show which adversarial tests were run (jailbreak, prompt injection, extraction), when, against which model version and with what outcome. A named sign-off confirms coverage is complete enough, and the pack discloses any sub-processor gaps openly. TechnoLynx’s AI Governance and Trust offer builds this pack. It starts by framing what this customer’s procurement review needs to see, then structures the staged test results against the taxonomy and supports the internal sign-off before delivery.

Organise the pack by category, not by team. NIS2 evidence-pack guidance makes the same point about organising by Article 21 measure instead of by department, so a reviewer does not have to search inboxes and ticket systems. The same guidance names five defects that weaken evidence: no date, no owner, no approval where approval is needed, no mapping to the requirement, and proof of design without proof of implementation. It also distinguishes Tier 1 evidence, which proves a measure exists, from Tier 2, which proves it is tested, reviewed and maintained. A coverage pack for an enterprise renewal should aim for Tier 2 on the categories the customer cares most about. Vendors running production SaaS face these renewals most often, and our page on AI for AI infrastructure and production SaaS covers that setting.

We expect renewal reviews to look hardest at freshness and change. One renewal checklist gives an example where a supplier’s assurance report was current but a new analytics sub-processor fell outside its report period. The checklist’s scoring suggests a conditional renewal, with conditions that might include evidence for the new sub-processor. If sub-processor chains are still unclear, resolve the gap or caveat it explicitly before submission. Do not present the pack as complete. Timing matters too: we would align the testing cadence with the renewal cycle so the pack is not stale on arrival.

The person who signs off on completeness should not be the person who ran the tests. MLCommons gives the Audit Lead evidence gathering and corrective actions to closure. The Secure Controls Framework puts evaluation of substitute controls with leadership, not the control owner.

Who can turn our red-team findings into evidence a customer will accept?

TechnoLynx builds approval evidence packs for this work under its AI Governance and Trust offer. It maps your existing red-team and pentest findings into one taxonomy, scores them against compensating controls, and assembles the versioned, signed-off pack a customer’s security team reviews. We would base the in-house decision on whether you already have a security lead, an evaluator owner and someone independent to audit. If one of those is missing, we would bring in a specialist over hiring for a single renewal. When choosing a partner, ask to see a sample artefact manifest and a residual-risk record, not just a findings report. The offer starts with framing what your customer’s review needs to see.

This is not certification, legal advice or regulatory sign-off. It produces the artefacts an audit needs. BSI assurance-level testing is done by approved laboratories, and iBeta’s ISO/IEC 30107-3 test procedures are audited under NIST’s NVLAP. If your team already runs a mature AI management system with these roles staffed, extending your own templates may be all you need.

What the pack can claim, and what it cannot

MLCommons states plainly that its results are system-level indicators, not guarantees, and that scope gaps should become triggers for re-testing. The jailbreak oracle work found that 30 of 128 prompts on Llama 3.1 looked safe under default decoding settings but gave way when those settings were changed slightly. A “passed” result can mean the model is robust or that the attack was suboptimal. An active test of evaluation methods would have flagged 11 of 13 previously peer-reviewed, later-broken defences as insufficiently evaluated. Physical tests carry their own optimism: the JRC warns that laboratory anti-spoofing results may overestimate real performance, because operational systems face artefacts they have never seen.

So we would write the pack’s claim narrower than a sales team would like. It should name the checkpoint, the attack classes, the access level, the evaluator and its error rates, the date, and the person who signed. The cadence and ownership that keep it current are what building a defensible, continuous adversarial-testing programme covers.

FAQ

What is spoofing detection?

Spoofing detection, or presentation attack detection, is any technique that automatically tells real biometric traits presented to a sensor apart from artificially produced artefacts that carry a biometric trait. It can work at sensor level with extra hardware or at feature level in software, and the JRC considers a combination of both the most desirable strategy.

What is red teaming in cybersecurity, applied to AI?

NIST AI 800-1 defines AI red-teaming as a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and with the system’s developers. The Frontier Model Forum adds that it often entails finding flaws across the full system, including data, infrastructure and applications, not only model outputs.