Our Take
Panich is right: vendor transparency theater won't move the needle with burned-out clinicians still scarred by EHR rollouts. AI earns credibility in medicine by killing paperwork first, clinical decision-support second.
Why it matters
Healthcare providers are deploying AI into clinical workflows at scale without establishing trust mechanisms that match physician liability and patient risk. The EHR precedent—where tech added burden instead of relief—is still the reference point for how doctors evaluate new systems.
Do this week
Chief Medical Officers: audit your current AI vendor contracts this week to confirm they document the specific data, guidelines, and chart details used in recommendations, not just confidence scores.
The credibility gap in clinical AI
Dr. Niki Panich, chief medical officer at Penguin AI and a practicing family physician, is calling for what she terms "glass box traceability" in medical AI systems. Rather than operating as black boxes, AI tools should expose their reasoning to the physicians using them.
Panich's core argument is straightforward: if hospitals expect doctors to act on AI recommendations, those doctors need to understand how the system arrived at them. A confidence score buried three clicks deep in software menus does not qualify as transparency at the point of care.
She describes a specific clinical scenario to illustrate the stakes. An AI system recommended placing a patient on a new anticoagulant medication. Embedded in that recommendation was a nursing note about a recent fall at home. The physician reviewing the discharge hadn't yet read that note. Without visibility into the AI's reasoning, she might have approved the plan. With it, she paused, called the family, and reconsidered the entire treatment approach.
Panich argues that AI should surface options and present context rather than dictate care paths. The physician, not the algorithm, owns the final decision. Healthcare organizations need to make this clear in policy and in practice.
Medicine doesn't fit in training data
Panich's second argument addresses why visibility matters beyond process: clinical decision-making involves context that training datasets almost never capture. A patient terrified of needles. A grandmother serving as primary caregiver who cannot afford to be sedated for a procedure on a specific day. A cultural belief about treatment that may never appear in a guideline.
These details shift the right treatment plan. An AI system optimizing for textbook answers will miss them unless a physician is present to intervene—and a physician cannot intervene if she doesn't know what the AI considered.
This mirrors a persistent problem in healthcare innovation: the EHR era delivered efficiency to administrators and billing departments, not to clinicians at the bedside. Doctors were asked to do more documentation, more clicking, more administrative work, all branded as progress. The memory of that burden shapes skepticism toward AI today.
Credibility comes from unglamorous work first
Panich's path forward is unglamorous but specific. AI in medicine will earn credibility by tackling the least glamorous and most draining parts of clinical work: prior authorizations, chart reviews, insurance denials and appeals. When AI proves itself first by removing administrative burden—by making the day shorter and the work lighter in ways clinicians can feel directly—trust for clinical decision support will follow naturally.
She also calls for concrete accountability on model performance. Bias audits. Transparent reporting on where models perform well and where they fall short. Broader and more representative training data. A physician needs to know not just what an AI recommends, but the conditions under which that recommendation is reliable.
The path to adoption is not replacement of doctors with algorithms, but tools that physicians can challenge, question, and trust. That trust is earned through visible work that reduces burden, not through marketing language about efficiency and precision.