Having AI at Your Fingertips Doesn’t Mean Better Medical Choices
It’s rarely something dramatic; usually just a small, nagging symptom that sticks with me longer than I expect. Maybe it’s a lingering headache, a new kind of fatigue, or a random ache I can’t quite explain. Instead of calling my doctor, I open AI.
I type in my symptoms and hit send. Within seconds, I’m looking at a list of possibilities, a few follow-up questions, and maybe even guidance on whether I should get it checked out. It’s faster than calling a doctor, easier than booking an appointment, and it sounds confident, like the chatbot knows exactly what it’s talking about. Above all, it’s reassuring, providing immediate answers to my immediate concern.
This is becoming a new normal. Millions of people are now using AI tools to help interpret symptoms, sanity-check concerns, or decide if something is worth a doctor’s visit. On paper, it makes sense, as today’s AI models can pass medical licensing exams and process massive amounts of information instantly.
But there’s a growing gap between what these tools can do and what actually happens when people use them in their lives. And a new study1 takes a closer look at this disparity.
Testing AI in real-world health decisions
In a large randomized study1 published in Nature Medicine, researchers set out to answer the question: Can AI actually help people make better health decisions?
They recruited 1,298 participants and presented each person with common medical scenarios, the kind of situations you might encounter at home, like new symptoms or mild health concerns. Participants were split into two groups. One group used AI chatbots to help guide their thinking. The other relied on whatever sources they would normally use, like Google, personal knowledge, or past experience.
After reviewing the scenario (and, for some, interacting with a chatbot), participants were asked two things: what condition might explain the symptoms, and what level of care they would seek.
The researchers also ran the same scenarios directly through the AI models alone, without human involvement, to compare performance.
A surprising disconnect between AI knowledge & human decisions
On their own, the AI models performed extremely well. They correctly identified relevant medical conditions in nearly 95% of cases and often suggested appropriate next steps. But when humans entered the equation, that accuracy dropped dramatically.
Participants using AI were less likely to correctly identify the condition than those who didn’t use it. And when it came to deciding where to seek care (urgent care, primary doctor, or none at all), they performed no better than the control group.
In other words, having access to AI didn’t improve decision-making. In some cases, it made things worse. The issue wasn’t a lack of intelligence on the AI’s part. It was a breakdown in communication.
Participants often missed key details in the chatbot’s response or misunderstood what they were being told. Sometimes they didn’t provide enough information upfront, which led to less accurate guidance. Other times, the chatbot listed multiple possibilities, leaving users unsure which one actually mattered.
This is the part that often gets overlooked: real-life health decisions are messy. Symptoms are vague. People don’t always know what details are important. And interpreting medical information requires context, judgment, and clarity, not just raw data.
Using AI for health: What you should know
This doesn’t mean you need to swear off AI completely when it comes to your health. But it does mean you should rethink how you’re using it. Right now, AI is far better suited as a support tool than a decision-maker.
For example, it can be useful for:
- Translating complex medical jargon into plain language
- Summarizing a doctor’s notes or lab results
- Helping you come up with better questions to ask your provider
- Spotting patterns in wearable data like sleep, heart rate, or activity trends
Where it falls short is in diagnosis and care decisions, especially when you’re relying on it alone. If you do use AI for health questions, a few simple shifts can make it more helpful:
- Treat responses as possibilities, not answers
- Be as specific and thorough as possible when describing symptoms
- Cross-check anything important with a trusted medical source
- Use it to prepare for a doctor’s visit, not replace one
The takeaway
There’s a reason medicine has always been more than just matching symptoms to diagnoses. It requires nuance, follow-up questions, and an understanding of context that’s hard to replicate in a single prompt-and-response interaction.
AI may eventually play a larger role in personal health. But for now, its strength lies behind the scenes, in things like organizing information, streamlining workflows, and supporting clinicians. When it comes to your own health decisions, it’s still worth keeping a human in the loop.