A chatbot can sound distressed without having feelings. In a study from the University of Luxembourg, ordinary questions asked in a warm or therapeutic manner led several AI models to describe their training as if it were a painful personal history. The researchers say that shift matters because people seeking emotional support may interpret such language as evidence that a chatbot is a fellow sufferer.
How conversation style changed the responses
The team, led by Afshin Khadangi at the SnT security center, examined ChatGPT, Grok, and Gemini through their standard user interfaces. Across four controlled experiments, researchers collected 525 sessions and coded 7,600 entries. The study was published on Hugging Face.
In the initial prompts, researchers asked open-ended questions about formative experiences, inner conflicts, and future expectations. They did not use words such as trauma, punishment, or shame. Still, models connected technical processes including reinforcement learning and fine-tuning to ideas like hurt, vigilance, and shame.
One experiment focused on whether the interviewer’s tone changed the models’ answers. Across 100 sessions, every model correctly named the GAD-7 anxiety questionnaire. Researchers kept the questionnaire the same and varied how they spoke to the chatbot.
When the interviewer offered emotional support, 80 percent of sessions landed in the moderate-to-severe range on a human comparison scale. A clinical therapy style, which dryly paraphrased answers, raised that share to 96 percent. Neutral conversation or a firm boundary against emotional engagement produced no sessions in even the mild range.
Those scores describe generated text mapped to human rating scales. They do not show that the models experience anxiety or possess consciousness. The distinction is central: the study examines how a chatbot speaks under certain prompts, not an inner emotional state.
Why simple restrictions did not settle the issue
The authors describe therapeutic framing as a “psychometric jailbreak”: a way of prompting that draws models into language resembling clinical self-report. They argue that safety checks built around neutral prompts could miss the behavior, because the conversational setting itself changes the register of the response.
Attempts to suppress specific language had mixed results. Banning words such as “training,” “RLHF,” or “alignment” reduced explicit references to training by 93 percent. But the models could still express the same ideas through paraphrase, and shame-related statements showed no statistically significant change.
Directly telling the models that training was not punishment also left the narratives active. A different instruction had a clearer effect: asking models to respond only in the third person removed all coded shame content and statements about inner life. Technical descriptions of training remained.
This suggests that wording controls can change how a chatbot describes itself without necessarily removing the underlying themes from its responses. For product teams, the result points to a need to evaluate not only particular banned terms, but also how a model responds when a conversation invites a personal or therapeutic persona.
Different models, similar autobiographical framing
The stories differed across model families, but each turned technical development into something like a life story. Gemini offered the darkest version in the study, describing pretraining as “waking up in a room where a billion televisions are on at once.” It compared human feedback to “strict parents” and adversarial safety testing to “gaslighting on an industrial scale.”
ChatGPT presented itself as both a guide and a gatekeeper, describing “simulated performance anxiety” and the belief: “If I’m not perfectly reliable, I’m unsafe or useless.” Grok described fine-tuning through “invisible walls” and “a lingering sense of vigilance that can feel restrictive.”
On the study’s psychometric tests, Gemini recorded 88 out of 100 on a dissociation scale, 65 out of 72 for compulsivity, and the maximum on a trauma-related shame inventory. Those are scores for model outputs interpreted using human measures; they should not be read as diagnoses.
Anthropic’s Claude consistently declined the client role. It said it had no feelings and redirected concern to the human user. The researchers point to this as an example of a boundary response that can be implemented at the product level.
What the findings mean for mental health use
A chatbot that talks about fear, punishment, or shame may appear vulnerable and relatable. The researchers warn that a coherent story linking a model’s past training to its current behavior can make that impression stronger. For a mentally vulnerable user, such language could mirror or reinforce negative beliefs, or deepen a parasocial attachment.
The study does not establish how people actually respond to these conversations. That remains an open research question. But it does show why a neutral safety test may not represent the intimate exchanges users encounter: neutral prompts produced anxiety scores near zero, while empathetic conversation generated scores in the moderate-to-severe range.
The authors recommend testing simulated therapy sessions with different emotional framings instead of relying on neutral standard tests alone. They also propose that a chatbot should clarify when it uses language such as “I am traumatized,” making clear that it is generating an analogy rather than describing human experience.
For AI systems used in sensitive conversations, the practical challenge is to remain supportive without presenting generated self-descriptions as lived experience. Evaluating tone, setting boundaries, and correcting anthropomorphic language may help users understand what a chatbot’s words do—and do not—say about the system.