ChatGPT may be better received than human advice writers when people judge the answer in front of them. But when asked who they would rather turn to, most people still choose a person.
That tension is the main finding from a study by researchers at the University of Melbourne and the University of Western Australia, published in Frontiers of Psychology. The research compared ChatGPT responses with human advice-column responses to social dilemma questions.
What the study compared
The researchers selected 50 social dilemma questions at random from ten popular advice columns. They then compared the columnists' answers with answers produced by the paid version of ChatGPT using GPT-4.
In the study, 404 subjects saw a question together with two answers: one from a columnist and one from ChatGPT. Participants were asked to judge which answer was more balanced, comprehensive, empathetic, helpful, and better overall.
The result was clear across the categories the study measured. ChatGPT was rated ahead of the human advisors, with preference rates ranging from about 70 to 85 percent in favor of the AI.
Why length was not the whole explanation
One obvious concern was that ChatGPT may have had an advantage because its answers were longer. Longer advice can seem more careful, more complete, or more emotionally attentive, especially when a dilemma is complicated.
The researchers tested that possibility in a second study. They shortened the ChatGPT responses so they were about the same length as the advice columnists' responses.
That second study still supported the first result, though the advantage was slightly lower. In other words, ChatGPT's stronger ratings were not only a result of giving more detailed answers.
People still prefer human advice
The most important twist is that better ratings did not translate into a stronger desire to receive advice from AI. Despite the positive judgments of ChatGPT's answers, 77% of participants said they preferred a human response to their social conflict questions.
The source article notes that this preference lines up with previous research. It also points to a key complication: participants could not reliably identify which responses came from ChatGPT and which came from humans.
That matters because it suggests the preference for human advice was not simply about the quality of the answer. The study points instead toward a social or cultural preference for hearing guidance from another person.
What this says about AI empathy
The findings fit into a broader pattern described in the source article. One earlier study by psychologists found that ChatGPT could describe possible emotional states in scenarios on the Levels of Emotional Awareness (LEAS) scale in much more detail than humans.
Another study of AI empathy, published in April 2023, found that people could perceive AI responses about medical diagnoses as more empathetic and higher quality than physician responses. However, that study did not examine whether the responses were accurate.
Taken together, these examples show a narrow but important point: people can rate AI-generated responses highly on qualities such as empathy, helpfulness, and detail. That does not mean they fully trust AI as a source of personal guidance.
The adoption gap
The study points to a practical challenge for AI advice tools. A response can be rated as balanced and useful while the person reading it still wants a human behind it.
The researchers suggest that future work should examine this reaction more closely. One proposed approach is to tell participants in advance which answers were written by AI and which were written by humans, then see whether that changes willingness to seek AI advice.
For now, the lesson is not that ChatGPT has replaced human judgment in sensitive situations. It is that people may value the substance of AI advice while still attaching special importance to human presence, especially when the issue involves social conflict.