Microsoft medical AI excels in test, not yet in clinics

Microsoft's experimental system, called MAI-DxO, achieved an outstanding result when processing complex medical cases. According to the published data, it outperformed the general physicians participating in the study in selecting the correct diagnosis, while also taking the cost of the required tests into account.

However, the result must be interpreted within the context of the test conditions. The system worked on previously documented, text-based cases rather than examining real patients, and the comparison did not involve physicians specialising in the individual diseases. For now, the technology is therefore a promising decision-support tool, not an autonomous digital diagnostician.

What did the comparison show?

According to a Chosun Health report, MAI-DxO achieved diagnostic accuracy of 83.2 per cent on a set of tasks based on complex cases from the New England Journal of Medicine. In the same comparison, general physicians who were allowed to use the internet and textbooks achieved 36.1 per cent.

The publication also reported another comparison related to emergency care. This involved 325 cases from the MIMIC-IV-ED database: the system achieved 80.3 per cent accuracy, while the physicians averaged 46.8 per cent.

The figures measured on the first set of tasks support the description ‘more than twice as accurate’. However, this applies only to that particular experiment and does not mean that the technology would demonstrate such an advantage for every disease, in every medical specialty or in every clinical situation.

Why was the previously reported gap smaller?

Earlier reports about the development highlighted a performance gap of up to fourfold. The figures should not be compared directly because the testing conditions differed.

In the earlier comparison presented in the original research paper, physicians were not allowed to use external reference materials. In the later test, they had access to the internet and medical literature, which improved human performance and therefore narrowed the gap between the two sides.

This is an important methodological detail. The value of a diagnostic technology is determined not only by the proportion of cases it solves correctly, but also by what assistance and data it used, and against whom it was compared.

It divides the diagnostic process into multiple tasks

MAI-DxO is not a simple question-and-answer system. According to the Microsoft Research overview and the research paper, it is an orchestrator architecture that divides the diagnostic process into multiple roles.

One component proposes possible explanations, another selects further tests, while separate components monitor costs and check the conclusions. Its operation can be compared to a virtual consultation among experts: rather than producing a final answer immediately, it gradually narrows the range of possible diagnoses as new information becomes available.

This structure may be important because diagnosis in practice is not simply a matter of recognising symptoms. A physician must make a series of interconnected decisions about which questions to ask, which tests to order and when enough information is available.

SDBench assesses decisions as a sequence of steps

An important element of the research is the Sequential Diagnosis Benchmark, or SDBench. Traditional medical tests often provide the complete case description and possible answers at the same time. This may be suitable for measuring factual knowledge, but it only approximates the real diagnostic process to a limited extent.

SDBench instead simulates a step-by-step decision-making process. The system being assessed must determine what information or test it still needs, then proceed on the basis of the new results. According to the research paper, the framework assesses both diagnostic accuracy and the cost of the tests used.

According to the Microsoft Research summary, diagnostic costs fell by 20–70 per cent, depending on the settings assessed. This result was obtained in a simulation, so it is not equivalent to a reduction of the same proportion in a hospital's actual expenditure.

How could it help overstretched healthcare systems?

Diagnostic decision support may be particularly useful in situations where many possible explanations and large amounts of information must be reviewed. Such a system could consistently list potential conditions, flag contradictions between data points and indicate what information is still missing before a decision can be made.

This could reduce the information-processing and administrative burden on physicians. The technology could organise test results and medical histories, while the treating physician could evaluate its suggestions based on knowledge of the patient's condition, the physical examination and the relevant clinical context.

The cost-aware approach may be valuable because it does not necessarily recommend carrying out every available test. Instead, it can rank the next steps according to how much new information they are expected to provide and the resources they require. However, the actual clinical benefit must be demonstrated in real patient-care settings.

The test is not the same as a real clinic

One fundamental limitation of the study is that it was based on retrospectively processed, text-based cases. In a clinic, a physician does more than read written data: they question the patient, observe their condition, perform a physical examination and may revise earlier conclusions in light of new information.

According to reports by TechRepublic and The Next Web, the comparison involved general physicians rather than specialists in the fields relevant to the complex cases. The result therefore does not establish that MAI-DxO would also be more accurate than specialists in the areas concerned.

The retrospective assessment also does not fully answer how the system would handle incomplete, erroneous or contradictory data arriving in real time. Its consistency of performance across different patient groups, institutions and healthcare systems must also be assessed separately.

Approval, data protection and liability

According to a BMJ analysis, the experimental system did not have clinical or regulatory approval. Safety studies and regulatory assessment are therefore required before it can be used in patient care; a favourable test result alone does not authorise its use in hospitals.

Before deployment, it will also be necessary to clarify how physicians can verify the system's conclusions, how patients' sensitive data will be protected and how liability would be determined if it made an incorrect recommendation. A confidently worded but incorrect answer could create unwarranted trust.

The realistic goal is therefore not to replace physicians automatically, but to create a transparent and verifiable decision-support tool. Clinical responsibility, communication with patients, empathy and professional judgement in uncertain situations cannot be inferred from a diagnostic test score.

A promising method, but not yet a finished solution

MAI-DxO's result is notable because it treats diagnosis not as a one-off problem, but as a process of gathering information and making decisions. Taking costs into account and dividing tasks among multiple roles may more closely reflect certain elements of clinical reasoning than a traditional multiple-choice test.

However, high accuracy does not yet demonstrate that the system is safe or suitable for general clinical use. That will require independent studies in real patient care, professional oversight and clear regulatory frameworks. The findings are not suitable for self-diagnosis and do not replace a medical examination or personalised professional advice.