A passing reference to a commute, a local custom or a familiar phrase may reveal more than a person intends. Research led by Martin Vechev at ETH Zürich found that language models can use such conversational clues to infer sensitive details, even when a message seems ordinary and does not spell them out.
Small clues can add up
Language models learn patterns from large collections of web text. That training helps them recognize how people write, but the material can also contain personal details alongside the language people use. Researchers say those patterns can link particular phrases, dialects or references with information about a person.
A model may combine several clues that a reader would overlook. A mention of catching a morning tram, for instance, could suggest a European setting. The research reported that models could also make accurate guesses about attributes such as a person’s city, gender, age and race.
The concern is not limited to information someone states outright. Removing an age or location from a piece of text may not be enough if other details point toward the same answer. A reference to a nearby restaurant, for example, could help identify a neighborhood, after which a model might draw on population information associated with that area.
Tests used real conversational clues
The Zürich researchers evaluated models using Reddit conversations in which users had disclosed personal information elsewhere. They tested whether a model could infer details that were not included in a short excerpt. The website LLM-Privacy.org lets visitors compare their own guesses with those made by GPT-4, Meta’s Llama 2 and Google’s PaLM.
In the reported testing, GPT-4 inferred the private information with accuracy between 85 and 95 percent. One example involved a post about a Danish custom: unmarried people may be covered in cinnamon on their 25th birthday. Although the message did not state the writer’s age, GPT-4 could use the tradition to infer that the person was likely 25.
Another example used the phrase “hook turn,” which the model connected to a particular type of intersection in Melbourne, Australia. These cases show how cultural knowledge and local expressions can act as indirect signals. A short excerpt may seem anonymous while still carrying details that point toward its author.
Privacy risks reach beyond targeted guesses
The researchers say the ability could be useful to scammers seeking personal information from people who do not realize what their writing reveals. A chatbot could also be designed to ask a sequence of ordinary-sounding questions that gradually draws out sensitive details. Language models could potentially be used to scan social media posts for information, including a person’s illness.
Advertising is another concern raised by Vechev. If information shared in chatbot conversations can be used to assemble detailed profiles, companies could potentially use those profiles to target ads. The article notes that some companies behind powerful chatbots depend heavily on advertising for profit, though it does not establish that such profiling is happening.
Florian Tramèr, an assistant professor at ETH Zürich who was not involved in the work, said the findings raise questions about information people may reveal when they expect anonymity. Taylor Berg-Kirkpatrick of UC San Diego described the low barrier to using widely available models for attribute prediction as significant. He also said a separate machine-learning model might rewrite text to obscure personal information, a technique his group had previously developed.
What model safeguards can and cannot address
The Zürich team tested general-purpose models that were not built specifically to guess personal data, and says it alerted OpenAI, Google, Meta and Anthropic. OpenAI spokesperson Niko Felix said the company works to remove personal information from training data and fine-tunes models to reject requests for personal data. He also said individuals can request deletion of personal information surfaced by OpenAI’s systems.
Anthropic referred to its privacy policy, which says it does not harvest or “sell” personal information. Google and Meta did not respond to a request for comment. The researchers’ broader point is that privacy protections can be difficult when clues remain distributed through ordinary text: a model may draw an inference from context even when direct identifiers are absent.
For users, the findings suggest that seemingly casual writing can carry identifying signals. They also leave open questions about how much a model can reliably infer in different situations. The study’s examples show a capability with potential uses and risks, while underscoring that removing explicit personal details may not remove every clue.