Medical AI is often discussed as if better models will automatically lead to better decisions. A new study by researchers at MIT and elsewhere points to a more complicated reality: the same AI assistance can help one user while pushing another toward overconfidence or error.
The work focused on skin disease diagnosis and compared how non-experts and primary care providers responded to AI predictions and different explainable AI methods. The central finding was clear. AI assistance generally improved diagnostic accuracy, but the explanation attached to the prediction mattered differently depending on the user’s level of medical knowledge.
AI explanations are not neutral
Explainable AI methods are meant to help users understand when a model should be trusted. In the study, researchers tested several ways of presenting AI support for dermatological disease detection.
Those approaches included an AI prediction and confidence level without an explanation, similar images used to reinforce a prediction, a heat map that highlighted important regions of an image, and an LLM explanation written in plain language.
In principle, each method gives the user more context. In practice, that context can change behavior. A user may treat the AI as one piece of evidence, or may use it as the starting point for their own answer. That distinction is especially important in medicine, where a plausible explanation can feel useful even when the model is wrong.
The researchers found that non-experts and clinicians did not use the same AI output in the same way. That is why a one-size-fits-all approach to medical AI assistance is unlikely to be the best strategy for disease diagnosis tools.
Non-experts improved, but often by deferring
Non-experts in the study were asked to decide whether an image of a skin mole was cancerous, both with and without explainable AI assistance. Across the tested explanation approaches, their accuracy improved.
That improvement came largely from reliance on the AI system. The tools were especially helpful in diagnosing non-cancerous moles. A fairness-constrained model designed to combat bias against darker skin tones also significantly improved accuracy and reduced diagnostic disparities based on skin tone.
But the same reliance created risk. When the model was wrong, non-experts were more likely to be pulled in the wrong direction. The effect was largest with LLM explanations, and users were more confident about wrong answers when an LLM supported them.
The study also found that non-experts trusted LLM-based explanations whether they were right or wrong. They even found explanations more convincing when they were vague or generic. That is a warning sign for consumer-facing AI health tools, because plain-language fluency can make an explanation feel authoritative without making it correct.
Clinicians reacted differently
Primary care providers received a more difficult task: providing a differential diagnosis of dermatological disease. Their response to AI support was different from the response of non-experts.
Clinicians were not tripped up by incorrect AI assistance. Among the tested approaches, they performed best when they received only the model’s prediction, without an accompanying explanation. LLM explanations boosted their accuracy the least.
The difference appears to come from how expert users evaluate the output. A clinician can compare the AI result with training and an existing diagnostic hypothesis. A non-expert may use the AI explanation to form the hypothesis in the first place.
That means explainability is not simply a feature that can be added to a medical AI interface and assumed to help everyone. For some users, an explanation may support critical review. For others, it may increase automation bias and make the model’s answer feel harder to challenge.
Timing shapes trust
The researchers also looked at when users saw the AI explanation. Timing mattered. If an explanation appeared before the user had made their own diagnosis, users tended to become more deferential to the model.
That finding suggests interface design is part of medical AI safety. The question is not only whether the model is good, or whether the explanation is readable. It is also whether the system encourages the user to think first or invites them to anchor on the AI answer.
The study found another important split between human and AI performance. AI systems outperformed humans when the disease presentation was subtle. Humans did much better when images included atypical symptoms or unrelated features.
Together, those findings argue for AI systems that help users reason, rather than systems that simply present a confident recommendation. In some cases, it may be more effective to ask users for a diagnostic hypothesis first, then show an AI-based suggestion that raises other possible conditions for consideration.
Designing for the person using the tool
Several FDA-approved AI interfaces are already being used to help clinicians identify skin conditions in medical images and streamline early diagnosis. At the same time, non-experts can use AI-powered search engines to attempt digital diagnosis from prompts, often with LLMs explaining predictions in simpler terms.
Those two settings involve different users, different expectations, and different risks. The MIT-led study shows why medical AI assistance should be designed around the person using it, not only around the model’s average performance.
For clinicians, a concise prediction may be enough, and additional explanation may add less value than expected. For non-experts, a friendly explanation may be persuasive in ways that are not always beneficial.
The broader lesson is that explainable AI in health care should not be judged only by whether it explains. It should be judged by whether it helps the user think more carefully. In skin disease diagnosis, the same explanation can support an expert and mislead a beginner. That makes presentation, timing, and user expertise central parts of medical AI design.