Microsoft’s Diagnostic AI Beats Generalists in a Simulation, Not in a Clinic

On 22 October Science published the peer-reviewed paper in which Microsoft says DxO — the Diagnostic Orchestrator announced as MAI-DxO — gets 83.2% of 304 New England Journal of Medicine cases right, against 36.1% for generalists. For emergency admissions the figures are 80.3% versus 46.8%. The paper is public, peer reviewed and explains its method. Read in full, the comparison is closer than the headline.

• Key Point: Not wholly new. Science received it on 4 April 2026 and accepted it on 10 August; Microsoft posted a preprint in June 2025 with the same title, fifteen authors, system and 304 NEJM cases. The published version adds peer review, 325 MIMIC-IV-ED emergency admissions using real notes instead of teaching vignettes, an open-book comparison, and GPT-5 runs repeated on GPT-5.5 without changing prompts or orchestration.
• Key Point: DxO is not a Microsoft medical model but model-agnostic orchestration software coordinating outside models, here OpenAI, and simulating complementary specialists.
• Key Point: The comparator was practising generalists, including primary-care and acute-care doctors, internists and emergency physicians. The paper gives averages but not participant numbers; the twenty-one doctors with median twelve years’ experience come from the preprint.
• Key Point: The 83.2% covers all 304 cases; physicians were scored on 56 reserved during development.

Closed book, open book, question count

In the closed-book branch physicians could not use search engines or language models and reached 22.3%; in the open-book branch they had those tools and reached 36.1%. Only 36.1% is set against 83.2%. The preprint gives two reasons: NEJM cases are online and would have been found, and the researchers wanted to measure physicians’ own diagnostic ability rather than indirectly evaluate tools.

Behaviour differs sharply. Physicians asked a mean 5.6 questions before the first test; DxO asked 25.5. The machine keeps asking where the human turns to the laboratory, and its advantage concentrates in history-dependent diagnoses such as rare exposures and travel infections.

“discriminating clues that would not be easily found in broad test batteries”

Simulated patient, AI judge, real costs

There were no patients: only text records, with no physical examination or imaging. The paper admits the sessions did not capture:

“multimodal data such as imaging, waveforms, or pathology slides that are central to real diagnostic workflows”

A Gatekeeper mediates every request and returns only clinically obtainable information; when data are absent it generates diagnostically neutral synthetic results so the gap does not signal the diagnosis. A third model, the Judge, scores against a physician-written five-level rubric; levels 4 and 5 count as correct, meaning management would not change.

Costs used CPT charges from the public Kaiser Permanente Washington chargemaster, consulted in April 2026, with history and examination treated as a fixed visit cost. In its most accurate setting DxO is pricier than the doctors: 83.2% at $5,431 versus 36.1% at $4,725. It wins on both at lower settings: 80.9% at $2,241 and 79.6% at $1,303. Microsoft is right about the cost-accuracy frontier, but “cheaper” depends on configuration, and the headline one is not cheaper.

What cannot be concluded

The authors are candid. Their introduction says:

“These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice.”

On case selection they warn:

“CPC cases may overrepresent diagnostically challenging presentations, and both exclude healthy or benign encounters, so the overall case mix and disease prevalence do not mirror those of routine clinical practice.”

No case was a healthy person. The preprint added that false-positive rates could not be measured. They also cite comparing DxO with individual physicians as a limitation, since real diagnosis involves multidisciplinary collaboration and referrals, and expect the gap to narrow against specialists or teams. The rubric judges disease-level correctness but, in their words, “does not assess therapeutic…”

Microsoft has produced a serious, peer-reviewed benchmark and a notable result. It has not produced proof that a machine out-diagnoses medicine. The headline rests on text-only teaching cases, a simulated patient, an AI grader, a smaller physician sample and the most expensive configuration. The next stage must be validation with real patients and real clinical teams, where the comparison will be harder and more informative.