The question that decides whether a healthcare LLM use case is buildable is not how good the model is. It is whether protected health information enters the call, who stays accountable for the output, and whether the surrounding controls — encryption, access, audit logging, retention — can be engineered to produce evidence a reviewer will accept. Model quality is roughly constant across your candidate list. Control obligations are not.
That asymmetry is why “chatgpt for healthcare” evaluations so often stall. A team runs sample cases through the model, judges the output credible, and concludes the model is good enough for healthcare. Then they treat patient messaging, documentation drafting, coding support and triage summarisation as one integration problem with one integration pattern. The first use case ships. The second one exposes the mistake.
How should a team evaluate “ChatGPT for healthcare”?
Evaluate the workflow, not the model. Each candidate use case gets scored on three workflow properties before anyone writes an adapter:
- Does PHI enter the request or the response? Not “could it in theory” — does the designed data path carry identifiers, dates, free-text notes, or anything re-identifiable.
- Is a named human accountable for the output before it reaches a patient or a record? A drafting aid where a clinician edits and signs is a different accountability structure from an automated summary that lands in a portal.
- Can you produce evidence after the fact? Who called the model, with what input, when, under what version, and what the human did with the answer.
A candidate use case is PHI-free only if the request payload, the response, and every log line along the path are free of identifiers — not merely if the clinical topic is generic. That is the single test most model-first evaluations skip, because it is a question about plumbing rather than about intelligence. Logging is where PHI-free designs usually leak: the prompt is scrubbed, the observability tooling captures the raw payload anyway.
De-identification helps, but it does not zero out the control burden. It shifts the burden from the model call to the de-identification step itself, which now needs its own validation, its own failure-mode analysis, and its own evidence that it works on your data distribution rather than on a public benchmark. Free-text clinical notes are the hard case. In our experience with regulated document pipelines, de-identification accuracy on structured fields and on narrative text are not the same problem, and treating them as one is a common source of late findings (observed across TechnoLynx engagements; not a published benchmark).
The three-tier triage
Sort every candidate into one of three tiers. The tier determines the engineering cost, the evidence obligation, and the shipping order.
| Tier | What it looks like | Controls required | Ship order |
|---|---|---|---|
| PHI-free | Internal drafting on generic content: policy summaries, literature triage, template generation, code and query support against synthetic or de-identified fixtures | Access control, prompt/response logging without PHI, model version pinning, human review before use | First — lowest control surface, fastest to evidence |
| PHI-scoped | Anything touching real patient data or patient-facing output: note drafting from a real encounter, coding support on live records, message drafting to a named patient | BAA-covered API path, encryption in transit and at rest, PHI-safe audit trail, retention and deletion controls, named accountable reviewer per output, de-identification validation where used | Second — build after the control substrate exists |
| Not suitable for a general-purpose LLM | Autonomous triage decisions, dosing calculations, diagnostic conclusions, anything where the output is the decision rather than an input to one | Not a control problem — an architecture problem. Requires a deterministic or validated component, with the LLM confined to a non-decisional role if used at all | Do not build as an LLM step |
The third tier is the one teams resist. It is not a statement about model capability; it is a statement about what evidence you can produce. A workflow where the model’s output is the decision cannot be made audit-ready by adding controls around it, because there is no human accountability step to log. The engineered alternative is usually to decompose: a validated rules or scoring component owns the decision, and the LLM handles retrieval, phrasing, or drafting around it, with the decision path traceable independently.
What changes when the use case becomes patient-facing
The jump from internal drafting support to anything patient-facing changes four things at once, which is why the “reuse the integration pattern” instinct fails so predictably.
The accountability chain becomes external. An internal draft that is wrong is caught by the person who asked for it. A patient-facing summary that is wrong has already left the building, so you need a review gate that is enforced in code, not by convention.
The evidence surface grows. You now have to show not just that the model was called correctly, but that a specific human approved this specific output at a specific time — and that the approved text is the text that was sent. Diffing the approved version against the delivered version is cheap to build early and expensive to retrofit.
Retention becomes a design constraint. Prompts and responses containing PHI are records. Where they live, how long they live, and how they are deleted on request are decisions that belong in the architecture, not in a later compliance pass.
Failure modes get user-visible. Rate limits, timeouts and refusals need a defined behaviour that does not leave a patient staring at a blank message. This is ordinary distributed-systems work, but it lands in the review scope because the surface is clinical.
Consumer chat interface versus API deployment
Moving from the consumer interface to an API-based deployment inside your own stack does not reduce the control obligation — it relocates it to somewhere you can actually engineer. Pasting a note into a chat window puts the data path outside your control and outside your logs. An API call from a service you own means you control the payload construction, the scrubbing, the logging, the retention, and the version pinning. That is strictly better for evidence, and strictly more work.
The practical consequence is that a use case which is unbuildable via the consumer interface can be buildable via an API path with a BAA in place, purely because the evidence trail becomes producible. The reverse is not true: an API path does not rescue a tier-three use case.
Vendor healthcare offerings — a dedicated health product, an enterprise tier, a BAA-covered endpoint — are inputs to this checklist, not answers to it. A BAA covers the vendor’s obligations for data they process. It says nothing about whether your logging captures PHI, whether your reviewer gate is enforced, or whether your de-identification step works. Run the vendor’s posture through the same per-use-case columns as everything else and note which rows it actually closes. Usually it closes the transport and processing rows and leaves the workflow rows entirely to you. The broader division of labour here — what the platform gives you versus what your engineering has to own — is the subject of our work on what actually ships in life-sciences AI, and it shows up across the whole life sciences engineering practice.
Pre-review evidence checklist (per use case)
Before an internal reviewer signs off, you should be able to produce, for each use case individually:
- A documented data path showing where PHI does and does not appear, including log sinks and error paths
- The pinned model and prompt-template versions in effect, with a change history
- An access record: which service identities and which humans can invoke this path
- An audit trail linking input, output, the accountable human, and the disposition of the output
- The retention and deletion policy for prompts and responses, and the mechanism that enforces it
- For de-identified paths: validation results for the de-identification step on your own data, not a vendor benchmark
- A defined fallback behaviour for model unavailability, refusal, or malformed output
Six of these seven items are workflow engineering. One concerns the model. That ratio is the whole argument.
Sequencing the portfolio
Teams that separate PHI-free from PHI-scoped use cases can ship the low-control tier first rather than blocking the entire programme behind one hard case. This is the measurable payoff of triage: the proportion of candidates scoped PHI-free, the number of control-coverage gaps found before build rather than at review, and the rework avoided by not re-architecting a PHI-touching step after a reviewer finding.
We see the same failure often enough to name it: a programme picks its most valuable use case first, discovers it is tier two or tier three, spends a quarter building control infrastructure, and ships nothing. Picking a tier-one use case first buys you a working integration, a logging pattern, a version-pinning discipline and an evidence template — all of which the tier-two build then inherits.
The per-use-case control obligations you identify during triage are also what a governance artefact has to record, so the triage output is not throwaway analysis. It is the first draft of the document a reviewer will read.
One thing remains genuinely unsettled: how much of the tier-three boundary is a permanent architectural fact and how much is a current-evidence limitation. If validated evaluation methods for decisional LLM output mature, some of that tier moves. Until then, the honest position is that the boundary is drawn by what you can prove, not by what the model can do.
Frequently Asked Questions
Which healthcare use cases can run on a general-purpose LLM without PHI entering the call, and which cannot? Internal drafting against generic or synthetic content — policy summaries, literature triage, template and query generation — can be genuinely PHI-free if the payload and the logs are both clean. Anything driven by a real encounter, a live record, or a named patient cannot be, because the input itself carries identifiers. The distinguishing test is the data path, not the clinical topic.
How do I scope a use case PHI-free, and does de-identification actually remove the control burden? Scope it by auditing every hop the payload takes, including observability sinks and error handlers, which is where PHI-free designs most often leak. De-identification does not remove the burden; it moves it onto the de-identification step, which then needs its own validation on your data distribution. Narrative clinical text is substantially harder to de-identify reliably than structured fields.
Which healthcare use cases should not run on a general-purpose LLM at all, and what are the engineered alternatives? Any workflow where the model’s output is the decision rather than an input to one — autonomous triage, dosing, diagnostic conclusions. No amount of surrounding control makes these audit-ready, because there is no human accountability step to record. The alternative is decomposition: a validated deterministic component owns the decision and the LLM is confined to retrieval, drafting or phrasing.
Is a general-purpose LLM ever usable for a clinician-facing workflow, or does clinician-facing always mean PHI-scoped controls? Clinician-facing is usable, but it lands in the PHI-scoped tier whenever the clinician’s input includes real patient data — which is most of the time. A clinician using the same tool for generic reference lookup with no patient context stays in the PHI-free tier. The user’s role does not set the tier; the payload does.
How do I evaluate a vendor’s healthcare-specific offering against the same per-use-case control checklist? Run the vendor’s posture through the checklist row by row and mark only the rows it genuinely closes. A BAA-covered endpoint or enterprise tier typically closes transport, processing and retention-on-their-side; it leaves your logging hygiene, reviewer gate, version pinning and evidence trail untouched. Treating the vendor’s compliance page as the answer is the most common way a review finding arrives late.
Three moves that make ChatGPT healthcare deployments stick
Start by isolating which workflows can tolerate probabilistic outputs and which cannot—everything else follows from that boundary. Revisit it when your workload shifts.
