The greatest risk of AI in healthcare isn’t that it makes mistakes, but that it makes them with absolute conviction.
In a clinical setting, a plausible error is far more dangerous than an obvious glitch. While a garbled sentence or wrong century gets caught in seconds, a fluent, well-formatted recommendation can easily slip past busy clinicians, entering the chart or training deck before anyone questions its logic.
The obvious glitch
A garbled sentence. A date in the wrong century. Caught in seconds — nobody acts on it.
The fluent error
A fluent, well-formatted recommendation. Slips past busy clinicians — into the chart or the training deck, before anyone questions its logic.
The glitch fails safe. The fluent error fails forward — into the medical record, the field rep’s presentation, and the prescription.
And once a clinician or executive signs off, the legal liability belongs to them.
This threat spans two distinct failure modes driven by the same underlying engine.
Failure mode 1: speech-to-text transcription
A 2024 Cornell University study (Careless Whisper: Speech-to-Text Hallucination Harms, Koenecke et al., ACM FAccT ’24) evaluated OpenAI’s Whisper — a foundation model integrated into clinical transcription tools used across roughly seven million visits. Researchers found that the model hallucinated entire phrases or sentences in about 1% of short audio segments.
The evidence
Koenecke et al., Careless Whisper: Speech-to-Text Hallucination Harms, ACM FAccT ’24.
Crucially, these fabrications occurred predominantly during quiet pauses — moments when the clinician was examining the patient rather than speaking. Worse, 38% of these hallucinations carried severe harm potential, including fabricated medications, violent remarks, and false clinical associations. A 1% failure rate sounds minor until it is applied across millions of patient encounters.
What this looks like in practice
A practice owner adopts an AI scribe. During an 11-second pause in a hypertension follow-up, the AI inserts two lines into the draft note: “ROS: Denies chest pain or dyspnea” and adds metoprolol to the active med list. Neither was discussed. The clinician quickly signs the clean-looking note.
The phantom metoprolol is now reconciled forward in perpetuity, and the unasked ROS question becomes legal exposure in discovery — signed under the doctor’s name.
Failure mode 2: large language model text generation
While transcription tools invent words during silence, generative LLMs invent evidence when asked for facts. A 2023 study published in Cureus revealed that when ChatGPT-3.5 was asked to provide medical references, 47% of the citations were entirely fabricated, and only 7% were both real and accurate. While newer models like GPT-4 improve on these numbers, fabrication rates remain significantly above zero.
Cureus, 2023. Newer models improve on these numbers — they do not reach zero.
What this looks like in practice
A commercial training lead uses a generic LLM to draft a clinical training module for field reps. The citations look authentic, complete with authors, journals, and DOIs. The deck clears MLR review. Months later, 200 field reps are pitching a clinical rule based on a paper that doesn’t exist, leading to off-label claims and compliance findings.
What this looks like in practice
A resident asks an ungrounded AI assistant for a dosing protocol on a late shift. The AI confidently suggests Cefepime 2 g every 8 hours for a patient on intermittent hemodialysis, formatting the answer precisely like the hospital’s internal guideline. The correct label dose for anuric hemodialysis patients is ~1 g every 24 hours post-dialysis.
The AI’s suggestion represents a 3× to 6× overdose, creating severe neurotoxicity risk.
The core technical flaw: predictability vs. verification
The root cause
Generic LLMs predict the next plausible word based on training patterns. They do not check facts against a source — so a fabrication arrives in the exact same tone and fluency as a verified clinical guideline.
Moving to safe clinical solutions requires a structural shift in system design.
01
Retrieval-Augmented Generation (RAG) & knowledge graphs
Medical-grade AI must retrieve, verify, and constrain every output against verified medical sources or peer-reviewed literature before generating a response.
02
Audio traceability
Clinical scribes must link every generated sentence in a note directly to the exact timestamp in the raw audio recording.
However, technical grounding is only part of the solution. The remaining risk requires procedural governance.
The governance & regulatory reality
To eliminate liability gaps, healthcare organizations must navigate an evolving global regulatory framework.
Under the 21st Century Cures Act and updated FDA Clinical Decision Support (CDS) guidance, software avoids strict medical device regulation only if clinicians can independently review the basis of the recommendation. If an AI tool cannot show its source, the clinician cannot satisfy this requirement.
Under EU MDR (Rule 11) and the EU AI Act’s human-oversight mandates, AI integrated into clinical decision-making carries strict compliance burdens.
Many AI scribe vendors automatically delete audio recordings after 24 hours to reduce HIPAA / GDPR Article 9 liabilities. While this mitigates data breach risks, it simultaneously destroys the audit trail. If a note is questioned, neither the doctor nor the compliance officer can verify what was actually said.
The 60-second vendor audit: 5 questions to ask before go-live
Before deploying any AI tool in a clinical or commercial workflow, ask your vendor to prove these five points live.
Show me the source
“Open the exact sentence in the source document behind this answer — or play back the raw audio line behind this sentence in the draft note.”
Audio retention & consent
“How long is raw audio retained, where is it stored, under what consent model, and is that explicitly stated in our contract?”
Local testing
“Was this system evaluated on our specific workflows, patient demographics, and accents, or merely on standardized industry benchmarks?”
Attestation & liability
“What does your accuracy disclaimer state, and what exactly is the user attesting to when they sign off?”
Post-market auditing
“What is the protocol for sampling and auditing outputs after deployment, and who owns that governance process?”
Fluency isn’t evidence
Key takeaway
Fluency must never be mistaken for clinical capability. If an AI tool cannot produce its source document or playback audio within 60 seconds, treat its output as an unverified search query — not a clinical result.
The future of healthcare AI does not belong to the most articulate chatbot, but to platforms engineered for verification, regulatory transparency, and meaningful human oversight.
The goal was never to replace clinical judgment, but to support it with evidence worth trusting.
Try it this week
Could you trace your AI tool’s last answer to its source — or play back the audio behind a line of the note — in under a minute?