Large language models may remember some books far better than others. A study by researchers at the University of California, Berkeley found that books with more online presence were more likely to be recalled by ChatGPT, GPT-4 and BERT. That pattern raises questions about copyright, but the researchers also point to a broader concern: popular stories may shape what these systems can say about culture.
Online visibility appears to shape recall
The team examined how well the models could reproduce or identify details from books. They used prompts with missing words and asked ChatGPT and GPT-4 to complete them. The results connected a book’s online frequency with how much detail a model retained.
That relationship matters because a model’s apparent familiarity with a book can affect what it can do with that text. The researchers found that models were more likely to handle tasks such as identifying characters or naming a publication year when a book was better known.
In other words, a strong answer about a popular novel may reflect exposure during training as much as a general ability to reason about books. Performance on familiar works therefore offers an incomplete picture of how well a model understands literature overall.
Familiar genres and titles stand out
OpenAI’s models showed particular strength with science fiction, fantasy and bestsellers, according to the study. The works named include 1984, Dracula and Frankenstein, along with Harry Potter and the Philosopher's Stone.
The comparison with BERT offered a clue about how this recall can arise. The researchers used BERT because its training data is known, and found that the dataset called “BookCorpus,” described as a collection of supposedly free books by unknown authors, included works by Dan Brown or Fifty Shades of Grey. BERT memorized information from these books because they were in its training data.
The example highlights why training material matters: if a book is included, a model may retain details that help it answer questions about that book. For models whose training data is not known, researchers have less direct evidence about which texts could explain a strong response.
Popularity can complicate cultural research
The study’s authors are not focused primarily on copyright. They are concerned that language models could become tools for cultural analysis while reflecting the uneven visibility of books in their training material.
Popular science fiction and fantasy often carry recurring narratives. If models learn those narratives especially well, their responses may emphasize ideas common in those works rather than the range of people’s experiences. That could influence research that uses model output to draw conclusions about culture.
The effect is not resolved by knowing that a model can discuss a famous book. The researchers say the difference in performance depending on whether a book was present in training material could introduce bias into cultural analysis. How that pattern affects model outputs and their usefulness for this work still requires further research.
Better tests require more transparency
The findings also challenge the use of popular books as a measure of model performance. A model that performs well on widely circulated titles may have benefited from extensive exposure to them. Such results may not indicate how it handles less familiar books.
The researchers call for disclosure of training data. Their analysis linking book popularity on the internet with memorization offers a rough guide, but they argue that understanding the underlying issue requires open models with known training data.
Copyright remains a separate question. Whether a model’s book recall becomes a legal issue may depend on how closely generated text matches material in its dataset, and the article says that question will have to be decided in court.
For now, the study’s practical message is twofold: training exposure can influence which books a model handles well, and that uneven recall should be considered when using language models to evaluate literary knowledge or study culture.