A recent study has shown that specialized medical artificial intelligence (AI) models can match or even surpass doctors in certain diagnostic and treatment decisions.
According to the Financial Times (FT), the scientific journal Nature published two studies on Wednesday showcasing the performance of Mira, an AI agent developed by researchers at the Technical University of Dresden and Heidelberg University Hospital in Germany, and Google’s AMIE. Both AI models outperformed human doctors in specific areas.
In the study, Mira processed over 500 emergency room clinical cases through chats with an AI agent playing the role of a patient. It was then evaluated on its overall medical capabilities, including diagnostic testing, medication prescriptions, and surgical scheduling.
The results revealed that Mira achieved higher diagnostic accuracy than doctors for diseases such as pancreatic cancer and pneumonia. Mira recorded an 87.1% diagnostic accuracy rate for eight conditions, including appendicitis and pulmonary embolism, surpassing the 78.1% achieved by a panel of six physicians.
Mira demonstrated performance equal to or better than a group of specialists and residents in terms of diagnostic accuracy and treatment quality. It also excelled in medication safety aspects, such as adherence to clinical guidelines and allergy checks.
AMIE, in simulated experiments with actors portraying patients, matched the overall patient management performance of 21 primary care physicians.
In another study, AMIE particularly excelled in test precision and alignment with clinical guidelines, outscoring human doctors. AMIE was praised for providing highly specific recommendations regarding dosage, treatment duration, and administration routes.
Google suggested that this indicates AI could eventually support medical practice, allowing doctors to dedicate more time to their patients.
However, scientists involved in both studies acknowledged the limitations of these AI models. The researchers explained that Mira still proposed suboptimal care for a small, but non-zero percentage of patients.
The research team also noted that while AMIE showed promising capabilities, it was not yet ready for real-world application. They emphasized the need for further research to address potential reasoning errors and other issues.
Experts pointed out that the cases introduced in this research might not fully reflect the complex and confusing medical situations that human doctors encounter in reality.
Julie Jacko, a chair professor of health informatics and data science at the University of Edinburgh, told the FT that overall, this is a strong experimental study and a meaningful advance, but it demonstrates performance against structured standards rather than fully capturing the complexities of real clinical decision-making.
Jakob Carter, a professor at the Technical University of Dresden who participated in the development of Mira, told the FT that he sees AI agents as similar to aircraft autopilot systems. These systems can assist and relieve healthcare professionals of routine tasks, but the ultimate responsibility will always remain with the doctors.