For the past decade, digital health focused on one primary problem: access.
Telemedicine showed up and changed the game by making it possible to consult a doctor without traveling, waiting in queues, or taking time off work. Healthcare, now in the hands of anyone with an internet connection, changed behaviour, expectations, and scale.
When patients seek medical help, they're looking for clarity, safety, recovery, and reassurance. Completing their consultation is what the organisation is optimising for. As telemedicine matured (and started to get called "telehealth"), solutions that solved for access began hitting their limit.
The best doctors are finite, and they're often overbooked. And the quality of care varies widely across providers, contexts, and time.
LLMs have stepped in to scale healthcare not by treating it as distribution problem but a consistency problem.
It is tempting to treat healthcare like other AI-adjacent industries: customer support, logistics, content moderation - automate common cases, escalate edge cases, optimise for efficiency.
In healthcare, one unsafe interaction can outweigh thousands of correct ones. Accuracy alone is not enough. The timing, tone, and knowing when not to proceed matters. An AI system that is “usually right” is not necessarily safe.
Healthcare is also governed by trust between
1. Patients and doctors,
2. Doctors and institutions, and
3. Institutions and regulators.
That trust is slow to build and easy to lose. An AI error is a breach of care, not just a bug.
The challenge of introducing AI into healthcare is threefold - design, governance, and responsibility.
A seductive question, no?
Healthcare, however, does not work that way. Like in other fields requiring astuteness, clinical judgment is not a single decision point but a chain of reasoning shaped by experience, intuition, and accountability.
A better framing for this would be:
Can AI be used in a way that doesn't erode safety, trust, and accountability?
It could if it isn't treated like a doctor.
While doctors remain accountable for diagnosis, prescriptions, and treatment decisions, AI supports the system by improving consistency, surfacing risk, and absorbing cognitive load where judgment is not the bottleneck.
The value of AI in healthcare lies in augmenting existing processes, not in clinical authority.
1. Creates consistency at scale
AI excels by being unwavering, not smarter.
It can ask the right follow-up questions, check for contraindications, and follow standard consultation structures. Time, fatigue, and volume force humans to take shortcuts, causing healthcare quality to degrade.
2. Detects signals early
A good AI system knows its limits. It recognises uncertainty, ambiguity, and risk signals early and routes them appropriately.
In healthcare, an escalation is not a failure.
3. Balances workload
By handling routine, low-risk interactions and documentation-heavy tasks, AI can free doctors to focus where judgment, empathy, and accountability matter most.
A safe AI system is one that:
1. Knows when a conversation is no longer appropriate to continue.
2. Treats escalation as a valid outcome, not an exception.
3. Is explicit about uncertainty without offloading responsibility to the patient.
Patients should experience safety mechanisms as care, and so these behaviours must feel intentional.
What the system refuses to answer, when it escalates, how it communicates uncertainty, and how it exits a conversation are all product decisions.
Speed, volume, and response accuracy are tempting metrics. However, optimising for any of these can incentivise unsafe behaviours.
Let's say we're optimising for speed. This is how the conversation could go.
I’ve been having chest discomfort since last night
Chest discomfort can be caused by acid reflux or muscle strain. Try resting and avoiding spicy food.
...
This response may be factually accurate in many cases, but it is unsafe.
The system responded before:
1. Asking about pain severity,
2. Radiation to arm or jaw,
3. Breathing difficulty,
4. Past cardiac history.
With healthcare systems, more meaningful signals tend to be subtler:
1. How often does a human need to override the system, and why?
2. Do patients return for the same unresolved issue?
3. Are outcomes improving, not just interactions completing?4. Does trust increase over time, or erode quietly?
A slightly slower system that asks clarifying questions first is clinically safer, even if it feels less “instant.”
Good healthcare AI will not feel magical. It will feel calm, consistent, and unremarkable. It will ask the right questions. It will stop when it should. And it will make it easier for clinicians to do what only they can do.
In an industry where trust is everything, guardrails are not a limitation. It is the point.
