Microsoft AI outdiagnoses general practitioners, but has yet to see a single patient

The cases come from the New England Journal of Medicine: patient files published as puzzles for physicians, often rare and tricky. Microsoft turned 304 of them into a question-and-answer game. The AI, like a physician, must question the patient, order tests and propose a diagnosis, with each test carrying a cost. On October 22, the journal Science published the results for this system, named MAI-DxO (for Microsoft AI Diagnostic Orchestrator).

The figure Microsoft highlights is striking: on these hard cases, MAI-DxO finds the correct diagnosis in about 83% of cases. General practitioners get there in 22% of cases without reference materials, and in 36% with their books and the internet. On more ordinary emergency room visits, drawn from a U.S. hospital database, the gap narrows: 80% for the AI, 47% for physicians, according to Microsoft.

A panel of virtual physicians

MAI-DxO is not a new model. It is a conductor: it has a large language model play several roles, like a panel of virtual physicians, one of whom is tasked with looking for alternative explanations. The published results rely on OpenAI's GPT-5. Microsoft says it reran the evaluation with GPT-5.5, without changing the method, and got results that were as good, or even better. The company also claims its system orders fewer tests, at a cost two to three times lower. But according to the figures in the paper, the configuration that reaches 83% costs about $5,431 in tests per case, more than physicians with reference materials ($4,725): the most economical configurations lose a few points of accuracy (79.6% at $1,303).

What the comparison does not prove

The authors themselves acknowledge that these results do not establish superiority over specialists, multidisciplinary teams or real-world practice. Several limitations weigh heavily:

• Files, not patients: the cases are text-only, with no physical examination, no images or samples to interpret, no vague medical history and no incomplete records.
• Cases chosen for their difficulty: no healthy patients, so no measurement of false alarms.
• An unbalanced comparison: the AI handled all 304 cases, the physicians a subset of 56 cases, and only general practitioners working alone were compared.
• Simulated costs, calculated from U.S. hospital prices, rather than observed savings.
• Authors who are all Microsoft employees: the paper was peer-reviewed, but no independent team has yet reproduced the results.

Microsoft itself writes that “the real test” will be performance in real clinical conditions. Nor is there any sign that the tool will be offered to physicians: Microsoft gives no release date.

Next step: the hospital

Microsoft announces a partnership with Beth Israel Deaconess Medical Center in Boston. Patients there will interact directly with the system, and its conclusions will be compared with the independent opinion of physicians. Neither the timeline, nor the number of patients, nor the date of the results is specified.

For a European healthcare facility, then, this is not a product to evaluate but a study to follow. Such a tool, if it came to Europe, would have to go through medical device regulation before entering a doctor's office. Microsoft is publishing the code of its benchmark, SDBench, for some of the cases: other researchers can therefore try to reproduce these figures, provided they obtain a license from the NEJM for the cases themselves.