Prof. Valmed AI Cuts Rheumatology Diagnosis Time but Lacks Accuracy

A recent randomized trial has revealed that Prof. Valmed, a large language model (LLM) certified for medical diagnosis by European regulators, can significantly reduce the time physicians take to diagnose rheumatologic conditions, though it does not enhance diagnostic accuracy compared to traditional methods. Conducted by a team led by Dr. Johannes Knitza at Philipps-Universität Marburg, the study highlights both the potential and limitations of AI in clinical settings, suggesting a need for caution in its application.

The trial involved 82 physicians from various medical disciplines in Germany and Norway, who were tasked with diagnosing three specific clinical scenarios, including Cogan syndrome and dermatomyositis. Physicians using Prof. Valmed averaged 94 seconds to reach a diagnosis, while those relying solely on their expertise took 206 seconds (P<0.001). Despite this efficiency, the accuracy rates were nearly identical: 33.3% for the intervention group versus 35.0% for the control group, a difference that lacked statistical significance. These findings raise critical questions about the role of AI in enhancing clinical decision-making.

Prof. Valmed, developed by Vera Roedel and Dr. Heinz Wiendl, is designed to assist rather than replace physicians, functioning as an "AI copilot." It received the European Union's CE mark in March 2025, signaling its readiness for market introduction. While the system aims to minimize the risk of "hallucinations"—false conclusions generated by AI—its performance in this trial suggests that the technology may not yet be ready to significantly improve diagnostic accuracy.

The trial's results also revealed a concerning trend: the use of Prof. Valmed appeared to inflate physicians' confidence in their diagnoses. Confidence ratings among the intervention group reached 57%, despite an accuracy of only 33%. This overconfidence could lead to misdiagnoses and underscores the need for better calibration of AI systems in medical contexts. The researchers noted that while AI may enhance perceived support quality and efficiency, it also raises safety concerns regarding overreliance on technology.

The findings come at a time when the healthcare industry is increasingly exploring AI applications to improve patient outcomes and streamline operations. While AI systems like Prof. Valmed can expedite the diagnostic process, the lack of improved accuracy signals that the technology may still be in its infancy. Companies developing AI for healthcare must prioritize not only speed but also accuracy and reliability to gain the trust of medical professionals.

As the market for AI in healthcare continues to expand, the implications of this trial are significant. Companies must navigate the balance between efficiency and accuracy while addressing the psychological impacts of AI-assisted decision-making. The trial suggests that while LLMs can assist in broadening differential diagnoses and improving workflow, their integration into routine care requires further validation through real-world trials.

Looking ahead, the healthcare sector may need to establish clearer guidelines for the use of AI in clinical settings. This includes developing training programs for physicians to better understand the strengths and limitations of AI tools like Prof. Valmed. As the technology evolves, fostering a culture of cautious integration—where AI is viewed as a supportive tool rather than a definitive authority—will be essential for maximizing its benefits while minimizing risks.