An artificial intelligence (AI) system designed to mimic a panel of medical experts – continually reviewing new information as it becomes available – outperformed participating human doctors in diagnosing clinical cases, while often reducing estimated testing costs. Previous evaluations of medical AI tools “have not tested diagnosis as an active, sequential information-gathering process that jointly considers diagnostic accuracy and diagnostic workup costs,” the study’s authors say. Clinical diagnosis is an iterative process in which physicians gather information from patient histories, physical examinations, and diagnostic tests to identify a medical condition. As new evidence emerges, they continually revise potential diagnoses and determine whether sufficient certainty exists to reach a conclusion or initiate treatment. Although language models (LMs) have demonstrated impressive medical reasoning abilities on written examinations and structured clinical cases, conventional evaluations typically provide complete case information upfront. “By compressing diagnosis into one inference step,” say the authors, “static benchmarks cannot measure whether a system asks the right questions, refrains from indiscriminate testing, updates the differential diagnosis as evidence accrues, or recognizes when sufficient certainty has been achieved to conclude the workup.” This could lead them to overstate diagnostic competence. To address this gap, Harsha Nori and colleagues developed the Sequential Diagnosis Benchmark (SDBench), transforming 629 clinical cases from The New England Journal of Medicine and emergency department encounters into interactive, step-by-step diagnostic challenges.

Using SDBench, Nori et al. evaluated Diagnostic Orchestrator (DxO), an AI system they created to guide diagnostic reasoning; it selects questions and tests with cost considerations in mind. Powered by GPT-5, DxO established a more favorable balance between diagnostic accuracy and testing costs than participating physicians across the evaluated cost-accuracy range. Depending on budget constraints, the system achieved accuracies of 79.6% to 84.5%, with average costs ranging from $1,303 to $8,945 per case. By comparison, physicians assessing curated clinical conference cases achieved average diagnostic accuracies of 22.3% without reference tools and 36.1% with them. Even the highest-performing physician, whose accuracy reached 51.8%, was surpassed by DxO in both accuracy and cost. The system also followed a distinct diagnostic strategy, gathering substantially more patient-history information before ordering its first test than physicians typically did. On routine emergency department cases, DxO achieved 80.3% accuracy at an average cost of $4,980 per case without budget constraints. Although these findings suggest that AI orchestration could support more efficient diagnosis, the authors argue that further evaluation is needed to establish its effectiveness in real-world clinical practice and its impact on patient outcomes. Morover, Nori et al. stress that the model cannot perform physical exams or observe patients directly. However, the system shows promise in supplementing a physician’s workflow.

For more insights, a “SciPak: Unpak’d” video interview with the lead and senior authors is available upon request from scipak@aaas.org.