No external certificate or benchmark score tells you that a prior-authorisation AI assistant is production-ready. Your organisation sets that bar and has to defend it. It takes the form of a milestone-based acceptance framework with explicit thresholds for accuracy, safety, latency, cost and regulatory evidence, a test harness shown to match your real ticket mix, and a recorded sign-off from named owners in compliance, procurement and operations.
TechnoLynx helps teams build that bar through its AI Readiness Scorecard. The engagement starts by working with compliance and procurement to turn their regulatory and business requirements into measurable production-readiness milestones for the prior-auth workflow.
No certificate, rubric or regulator sets the pass mark
Does a regulator’s clearance or a published framework settle the readiness question? The evidence points the other way. An audit of 519 FDA summary reports for cleared AI/ML devices found that 24.7% gave no evidence for any of the six FUTURE-AI trustworthiness principles, and that no report covered all six. Traceability was documented in 8.3% of reports and explainability in 3.5%. The authors conclude that regulatory approval alone should not be treated as a proxy for trustworthiness. Their study measures what the public reports disclose, not how the devices perform, so a missing principle may mean either non-compliance or thin reporting. For a buyer, that caveat leads to the same place: clearance documents do not give you the evidence you need to decide.
The voluntary frameworks give you structure, but none of them sets a threshold. The NIST AI Risk Management Framework is voluntary and does not prescribe risk tolerance. Where a sector has no established guideline, NIST says the organisation should define a reasonable tolerance itself. NIST’s Generative AI Profile goes one step further and suggests establishing minimum performance or assurance thresholds as part of go/no-go deployment approval. ONC’s HTI-1 decision support criterion defines disclosure fields with no required measures, baselines or thresholds, and it has no test tool or test procedure for certification. The NAIC model bulletin for insurers explicitly does not prescribe practices or documentation requirements. TRIPOD-LLM is a reporting checklist, and its authors state that it is not a quality appraisal tool.
Those instruments tell you what to document and how to organise risk work, and they leave the pass mark to the organisation deploying the assistant.
Does a prior-auth AI assistant have to clear a healthcare regulatory bar, and who checks?
Some stages of the pipeline are regulated and much of the engineering around them is not. A payer-side tool that helps decide coverage sits inside medical-necessity and utilisation-management rules. A provider-side tool that drafts requests touches HIPAA when it handles PHI, and its output must carry the elements payer systems require for processing. The first check should be whether each specific stage falls under a requirement at all.
| Layer | What it binds in a prior-auth pipeline | Who checks |
|---|---|---|
| Medicare Advantage medical-necessity rules (42 CFR 422.101(c)) | An algorithm may assist, but the determination must rest on the individual patient’s history, physician recommendations and clinical notes, and a physician or other appropriate professional must review adverse decisions before they are issued | CMS program audits, using physician reviewers on denied requests |
| MA utilisation management committee (42 CFR 422.137) | Prior-auth policies need committee approval from 1 January 2024, with the reasons for decisions documented in writing | A committee led by the plan’s medical director; documentation goes to CMS on request |
| CMS 2024 prior-auth final rule (CMS-0057-F) | Decisions within seven calendar days for standard requests and 72 hours for expedited ones, plus a specific reason for each denial; provisions apply from 1 January 2026 and most API requirements from 1 January 2027 | CMS, the issuing agency |
| State AI laws on utilisation review | Depending on the state: human review of denials, decisions based on individual clinical history, disclosure of AI use, periodic accuracy reviews and regulator audit access | State insurance regulators; in Texas the commissioner may audit an automated decision system at any time |
| NAIC model bulletin | A written programme governing AI systems that make or support regulated insurance decisions; at least 25 states had issued guidance based on it as of early April 2026 | State insurance departments, during investigations and market conduct actions |
| FDA clinical decision support boundary | Software is non-device CDS only if it meets all four statutory criteria, including letting the professional independently review the basis for the recommendation | FDA |
| ONC HTI-1 decision support criterion | Applies only to predictive DSIs supplied by certified health IT developers, not to tools a health system or third party builds itself | ONC-authorised certification bodies, through developer attestation |
| HIPAA | Any stage that processes protected health information, including vendor processing under a business associate agreement | HHS |
The table has gaps you should know about. Current federal Medicaid regulations do not directly address automation in prior authorisation. State oversight varies widely: California regulates AI tools in prior authorisation, while Tennessee and Washington have set up a council and a task force. FDA’s 2026 CDS guidance says nothing about AI, which leaves generative clinician-facing tools without a direct answer. When a stage’s status is unclear, record the open question in the evidence pack and do not quietly assume an answer in either direction.
What the evidence pack holds and what to require from the vendor
The evidence pack is an engineering output. It is the material that a compliance owner, auditor or procurement lead can read, defend and sign against, and it is produced as engineering work, not as policy material. The CHAI Applied Model Card was designed partly for this use, as transparency documentation within procurement processes. The ONC source attributes give a field list a vendor will recognise.
Ask the vendor for these, as a minimum:
- Intended use and decision role: whether the output informs, augments or replaces human judgement, and where a human must be in the loop.
- Out-of-scope uses and known limitations: the tasks, situations and populations where the tool should not be applied.
- Training data relevance: inclusion and exclusion criteria, demographic representativeness, and how relevant the data is to your deployment setting.
- Internal and external validation: validity and fairness on same-source and external data, the data source used for external testing, and the party that ran it.
- Subgroup fairness: results across subgroups such as race, ethnicity, sex, age and disability, with a note on what was not tested and why. CMS says MA organisations should, before implementing a tool, ensure it does not perpetuate or introduce bias.
- Model and prompt provenance: model version, any fine-tuning, prompt engineering methods, and whether the model was frozen or changing during evaluation.
- Monitoring and update process: how often validity and fairness are checked on local data, and how quickly performance is corrected when a risk appears.
- Revision triggers: a revised card whenever intended use changes, a new risk or adverse event is found, key metrics shift significantly or training changes structurally, with an updated card requested at least once a year.
- Stage-level logs: audit logs that attribute a failure to a specific pipeline stage, as one deployed clinical AI agent’s logs do.
- Third-party incident response: defined ownership and a rehearsed response plan for incidents in the vendor’s generative AI components.
In the FDA audit, traceability was one of the two most pronounced reporting gaps, documented in only 8.3% of reports. We would therefore ask for provenance and logging evidence by name and not wait for it to arrive with the performance tables. Turning these artefacts into an auditable trail, from model sign-off down to field-level configuration, is the subject of our companion piece on building evidence packs that separate audit workflows, approval gates and field mapping.
An AI readiness assessment framework that weighs accuracy, latency, cost and regulation
Do not use one blended score for both. A weighted matrix is the right tool for comparing vendors because it makes trade-offs explicit. As a go-live gate it has a weakness: strong accuracy and cost scores can average away weak regulatory evidence if the weights are not set carefully.
Google’s ML Test Score shows a way around this. It scores four sections separately and takes the minimum, not the sum, because a system has to address every section to raise its score. In a study of mental-health AI agents, all models scored close to perfectly on surface acknowledgement while therapeutic appropriateness collapsed to 0.22–0.33 at the highest severity for three of four models. That finding comes from a different clinical domain, but the mechanism carries over: an aggregate score can hide failures that are concentrated in the cases that matter most.
| Dimension | Role in vendor scoring | Role at the go-live gate |
|---|---|---|
| Clinical and administrative accuracy | Weighted score | A floor for each stratum (payer, specialty, request type), not an overall average |
| Severity-critical errors (wrong patient data, unsupported denial rationale) | Not weighted | An explicit ceiling, with gray-zone cases routed to human review |
| Latency | Weighted | A ceiling set by reviewer workflow and queue throughput, measured on the target serving hardware |
| Cost per ticket (inference plus human review time) | Weighted | A budget ceiling at forecast volume |
| Regulatory evidence | Pass/fail, never weighted | Every required artefact present and verified by a named, independent owner |
The milestones are where you encode NIST’s go/no-go determination. The deployed system has to be shown to be valid and reliable under conditions similar to deployment, its residual negative risk has to stay within the tolerance you defined, and it has to fail safely. A measured example of this kind of gating comes from a deployed EHR-embedded clinical documentation agent. No engineering change was deployed without quantitative evidence from version-to-version rubric comparison. The same programme reported a median processing time of 8.1 seconds per audio segment and cut inference costs by 20–30% with no measurable quality loss on its rubric. The point to take from it is that accuracy, latency and cost were tracked within one experimental framework, so each trade-off was visible.
How do we show the test harness reflects our real prior-auth tickets?
Our recommendation: document the harness’s coverage against a stratification of your actual ticket population, and score payer-required administrative elements separately from clinical content. NIST asks that accuracy measurements always be paired with realistic test sets representative of expected use, with the test methodology documented.
A measured prior-auth letter study shows why administrative coverage gets its own line. Researchers generated 135 prior-auth letters from three commercial LLMs across 45 synthetic scenarios in five specialties. Clinical content was strong, with mean rubric scores above 97%. CPT/HCPCS billing codes, however, were missing from all 45 GPT-4o letters and present in only 22 and 23 of 45 for the other two models. Authorisation duration appeared in none of 45 letters for two models and in 12 of 45 for the third. The authors identify six categories of administrative content that all three models consistently under-generated.
The same study names a gray zone. Four scenarios whose approval criteria are recent, inconsistently published or left to insurer discretion got imperfect scores from all three models, and the authors recommend flagging these for clinician review instead of generating a letter with default framing. They are also clear about the limits. The scenarios were synthetic, letter quality was scored rather than approval outcomes, and the structured prompts probably understate hallucination risk compared with incomplete real records. A separate deployment found that synthetic cases help cover rare scenarios but cannot yet fully replace real encounters.
Procedure for evidencing harness coverage
- Draw a sample of historical tickets and record their distribution by payer, specialty, request type, standard versus expedited urgency, and step-therapy involvement.
- Compare the harness’s distribution to that sample stratum by stratum, and list every stratum that is under-represented or missing.
- Score administrative fields (billing codes, authorisation period, place of service, follow-up plan) separately from clinical accuracy, using the payers’ own submission requirements as ground truth.
- Add gray-zone cases whose expected correct outcome is “route to human review”, so the harness tests whether the assistant knows when to stop.
- Mark which cases are synthetic and report results for real and synthetic cases separately.
- Version the harness and attach the coverage report and gap list to the sign-off record.
Who signs off the bar and the coverage, and how it gets recorded
Each owner signs for the evidence they can actually verify. Compliance signs for regulatory evidence. Procurement signs for cost and performance, including the contracts and SLAs that should specify quality standards and security requirements. Operations signs for harness coverage and post-deployment monitoring. On the payer side for Medicare Advantage, prior-auth policies also pass through the utilisation management committee led by the medical director. Above them, NIST expects executive leadership to take responsibility for decisions about AI deployment risk.
Independence has to be written down, not assumed. The NAIC bulletin asks insurers to address the independence of decision-makers and lines of defence at each stage of the AI life cycle. NIST’s generative AI guidance calls for regular assessments by internal experts who were not front-line developers of the system, or by independent assessors. Two vendor-commissioned audits of AI hiring tools show how independence erodes: in them, the audited company helped frame the questions or controlled the scope and publication of results. For each milestone, the record should therefore name who verified it, their relationship to the vendor and the build team, and what access they had.
The sign-off record should hold the acceptance thresholds and the rationale for each, the harness version and coverage report, the evidence pack version, the documented residual risks, any dissenting views, and the revision triggers that reopen the decision. The UM committee’s written-reasons requirement is a good model for the level of detail. A sign-off that cannot be traced back to specific artefacts gives an auditor nothing to examine.
Who can help us define production-readiness and build the evidence pack for our prior-auth assistant?
Through its AI Readiness Scorecard, TechnoLynx designs the acceptance framework and produces the engineering evidence behind it, so your compliance and procurement owners have something concrete to sign. It sets weighted scoring criteria and acceptance thresholds for accuracy, latency, hardware cost and regulatory evidence, tailored to your prior-auth workflow. TechnoLynx then integrates that framework into the release process, runs evaluation benchmarks against those milestones on your target deployment hardware, and supports a staged rollout with sign-off gates and ongoing monitoring. TechnoLynx delivers the governance side under AI Governance and Trust, part of its wider AI engineering work for life sciences and healthcare.
TechnoLynx builds this evidence as engineering, not as policy advice or certification. A team that already runs disciplined model evaluation and has clear regulatory ownership can build this in-house using the instruments above. The TRIPOD-LLM authors expect assurance labs such as the Coalition for Health AI to play a role in verifying and maintaining clinical AI model cards. Where independence from both vendor and buyer matters most, use an external auditor with no employment or contractual ties to either side.
The milestone a blended score cannot see
The hardest thing to evidence is not the accuracy figure. It is the assistant’s behaviour on requests whose approval criteria are recent, variably published or left to insurer discretion, where all three models in the prior-auth study scored imperfectly on the four such scenarios. A readiness bar that rewards the assistant for always producing an answer will pass a system that fails exactly where denials are most contested. Write “correctly routes to a human” into the milestones as a success condition, test it with cases built for that purpose, and have someone outside the build team sign for it. That one item is a fair test of whether the rest of the framework has teeth.
FAQ
What are the four steps of the NIST AI risk management framework?
NIST calls them functions rather than steps: GOVERN, MAP, MEASURE and MANAGE. GOVERN runs across the whole life cycle, while the other three are applied to specific systems and stages.
Which regulatory agency is responsible for ensuring the safety and efficacy of medical devices?
In the United States it is the Food and Drug Administration. For clinical decision support software, FDA’s guidance sets out which functions fall outside the device definition.
