By MANUEL PINILLA
Every few months a new study reports that an AI model has outperformed doctors on a set of diagnostic cases. The latest one I’ve read came out in Science in October: Microsoft’s MAI-DxO, tested on 304 complex cases from the New England Journal of Medicine and run as step-by-step simulations, reached the correct diagnosis in about 83% of them. Primary care doctors and general internists, working on their own with reference materials, got 36%. Microsoft says the next step is a study with real patients.
I build clinical AI, so these results matter to my work, but they answer a different question from mine. In the setting we work on, emergency departments and primary care, a doctor will be in the room, and that doctor signs the diagnosis and answers for it. While that is the case, a score of AI against doctors tells me little about whether a tool helps. What I need to know is whether the doctor, with the AI, gets the diagnosis right more often, and what happens when the AI is wrong.
There is a second gap. A published case arrives already written up: the history is complete, the relevant findings are there and the question is clear. A real patient tells the story out of order, leaves out what they don’t think matters and sometimes can’t put into words what they feel. Getting from that to a good differential is a large part of a doctor’s job, and a benchmark mostly skips it. How a tool behaves in a specific setting, with the information that setting really provides, matters as much as what the model knows.
Where the AI shows up in the consultation
I’m an engineer, not a physician, and when I think about these tools my first concern is the interaction more than the model. To use a chatbot during a consultation, a doctor has to stop, summarize the case, write a question, read the answer and judge how far to trust it before turning back to the patient. In a ten-minute primary care visit or a crowded emergency department there is rarely time for that, so in practice the tool tends to be consulted afterwards, to check a diagnosis that is already more or less settled. By then the patient has left, and with them the chance to ask a different question or examine something that was missed.
That is the problem we set out to work on at Medsys AI, a startup in Madrid. Our assistant listens to the consultation and, as it goes on, suggests diagnostic hypotheses, questions worth asking and things to check in the physical examination. The suggestions are updated as new information comes up, and the doctor doesn’t have to type anything. What to do with them is the doctor’s call. The picture I had in mind was an attending standing behind a resident who is unsure, available to doctors at any stage of their career.
A couple of design decisions follow from that. The assistant shows several hypotheses with the reasoning behind each, drawn from more than a thousand clinical guidelines and checked by more than one model before they reach the screen. And the system is built so that we can change the underlying model and rerun our evaluations when a better one appears.
Google and Beth Israel Deaconess published a study in The Lancet in which about a hundred patients with an upcoming primary care visit for a new problem chatted with Google’s AMIE beforehand, with a physician supervising every conversation live. The doctors received a transcript and a summary before seeing the patient, and none of the conversations had to be stopped for safety. The physicians said it moved the visit from collecting information towards checking it and deciding with the patient. It is a different design from ours, before the consultation instead of during it, and I think it is a good example of how to do this: the AI does part of the work and the doctor keeps the decision.
What we measured
We evaluated our assistant with doctors using it. Thirteen board-certified family physicians in Spain conducted 260 voice consultations with virtual patients, played by a voice model held to the details of each case. There were 40 cases, chosen for their diagnostic difficulty, such as a pulmonary embolism or an autoimmune disease presenting with vague symptoms. Each case was randomly assigned to be done with or without the assistant, and three adjudicators who didn’t know which arm a consultation came from scored the diagnoses. The results were published in JMIR Formative Research.
The correct diagnosis was among the doctor’s top three in 62.3% of unassisted consultations and in 74.6% of assisted ones, a difference of 12.3 percentage points. Counted as misses, that is 32.6% fewer, and 44.8% fewer in the hardest cases. Physicians rated the tool 4.38 out of 5 for usefulness.
The study has clear limits. The patients were simulated, thirteen doctors is a small sample, and in the unassisted arm physicians had no external resources (no UpToDate, no search engine), which is not how they normally work. These are preclinical results. We are now running a clinical trial with real patients in the emergency department of a major hospital in Madrid, authorized by a research ethics committee and by the Spanish Agency of Medicines and Medical Devices.
Two (out of the, probably, many) questions we haven’t answered
The first is time. Assisted consultations took about 35 seconds longer, roughly 10.7%. In an emergency department with a full waiting room, 35 seconds per patient adds up over a shift. We don’t know yet where the acceptable limit is, or how much of that time goes into questions that change the diagnosis and how much into reading suggestions that don’t.
The second is overreliance. We also looked at what happened in the consultations where the AI’s diagnostic suggestion was wrong. The system generated suggestions in both arms, but only the assisted physicians could see them. In those consultations with wrong suggestions, assisted physicians submitted the same wrong diagnosis 55.8% of the time, compared with 38.8% for unassisted physicians, who never saw it. The difference was not statistically significant, so the study does not show that the assistant pulled doctors toward wrong answers. It doesn’t rule it out either, and with a gap of that size in a small study, I read it as something to follow closely. Showing several hypotheses with their reasoning doesn’t by itself stop a doctor from anchoring on the first one. The clinical trial measures this again, with real patients.
Why diagnosis
Most of the AI investment in healthcare over the last two years has gone into ambient scribes. They give doctors time back, and an imperfect note carries relatively little risk, so that makes sense. We see our work as complementary: a scribe writes up the conversation, and we try to help while it is still happening.
Diagnostic error is where a large share of the harm is. AHRQ’s 2022 review estimated that about 1 in 18 emergency department patients in the US is misdiagnosed, around 7.4 million people a year. Emergency medicine societies disputed how that figure was extrapolated, and the real number may well be lower. Even a fraction of it is a large problem, and documentation tools do little about it.
The regulatory path
In Europe, software that contributes to a diagnosis is a medical device, and not in the lowest risk class. To get CE marking we need a technical file covering benefits and risks, assessed by a notified body. For us that realistically means some time in 2027. It is also the process that has us running a trial with real patients and reporting risk measures like the one above.
What I’d like to see measured
For the kind of system we are building, standalone model performance is not enough to know whether it helps. I’d like to see more studies, ours included, that measure whether doctors using the AI are more accurate, whether they are safer when the AI is wrong, and how much time it costs them, first in simulation and then with real patients.
Even that is a step on the way. A correct diagnosis matters because of what it does for the patient, so the measure that counts in the end is how patients do afterwards: over the following days in an emergency department, over months or years in primary care, depending on the specialty. We keep that in mind when we design the assistant and its evaluations. We have a first piece of evidence that this approach can work, and a clinical trial underway to see whether it holds up. I’ll share what we find.
Manuel Pinilla is co-founder of Medsys AI, a Madrid-based startup building real-time AI support for physicians during the consultation, and a co-author of the study. The paper with the preclinical results.
