Artificial intelligence is taking on a growing role in health care, from helping hospitals sort patients by need to informing diagnoses and treatment plans. These tools may offer useful information, but they can also influence who gets heard when care decisions are made. If clinicians treat an algorithm’s output as the final word, patients’ experiences and preferences may count for less.
When an algorithm takes authority
In medicine, paternalism describes the belief that a doctor knows best and should decide what happens to a patient, even when the patient’s feelings, beliefs, or culture point in another direction. AI can reinforce that pattern if its recommendations are treated as the strongest form of evidence.
Researchers Melissa McCradden and Roxanne Kirsch describe this risk as algorithmic paternalism. The concern is not simply that a computer might make a recommendation. It is that a clinician might give that recommendation greater weight than a patient’s own account or the clinician’s judgment.
That risk has a practical basis. In a study published a few years ago, oncologists compared their skin cancer diagnoses with an AI system’s conclusions. Many accepted the system’s results even when those results conflicted with their own clinical opinions.
AI predictions reflect the data behind them
AI systems learn from information collected in the past. That information can be limited, biased, or wrong, and a system trained on it may not perform equally well for everyone. The source article notes that some tools work less well for women and people of color than for white men.
A system that identifies patterns in skin cell biopsies, for example, may learn from earlier cancer diagnoses. But if clinicians were more likely to miss cases in people of color, the historical data may carry that gap forward. Recognizing patterns associated with past decisions does not guarantee that the system will serve every patient well.
As Sandra Wachter, a professor of technology and regulation at the University of Oxford in the UK, explains, an algorithm is not trained in the way a doctor learns about the human body and illness. McCradden and Kirsch make a related point: AI can predict, but it cannot understand. A prediction can inform care without capturing everything that matters to the person receiving it.
Better data takes deliberate work
One proposed way to improve AI systems is to train them with more diverse biological information and information about the beliefs and wishes of different communities. That would require researchers and designers to collect data that may be missing now.
Collecting it is expensive, which may conflict with the aim of using AI to reduce health-care costs. Still, the quality and range of data matter: a tool that performs well for one group may not work as well for another, whether differences relate to biology or beliefs.
Designers also need to consider the people who will be assessed by their systems. That means asking whose experiences are represented in the data and whether a tool’s recommendations make sense across different communities. Better data can address some weaknesses, but it does not make an algorithm a substitute for a patient’s perspective.
Use AI as one input to care
A useful comparison is the role of X-rays and MRIs. These technologies can help inform a diagnosis alongside other health information; their results are part of a wider decision. AI can be used in a similar way, as support for clinical judgment rather than an authority that overrides it.
That approach leaves room for doctors and patients to collaborate on treatment decisions. Patients should be able to choose whether they want a scan and what they want to do with the results. The same principle applies as AI enters care: its growing use need not mean that patients give up control over decisions affecting them.
AI may help identify patterns and guide care, but its output is shaped by the information it has learned from. Keeping that limitation in view can help clinicians weigh recommendations alongside their own judgment and the patient’s lived experience. The goal is to use the technology without letting it decide whose knowledge matters.