Large language models have long reflected harmful material found in the internet text used to train them. Developers have tried to reduce that toxicity, but new research suggests a difficult problem remains: some racist associations may become less obvious while staying active beneath the surface.
The study examined how five AI models, including OpenAI’s GPT-4 and older models from Facebook and Google, responded to speakers using African-American English, or AAE. The researchers found that the models made more negative judgments about AAE speakers than speakers using Standard American English, or SAE, even when the sentence meanings were the same.
What the study tested
The researchers did not tell the models the race of the speaker. Instead, they asked the models to evaluate language written in AAE and SAE. That design matters because it tested whether the models would infer social judgments from dialect alone.
The results were stark. The models were more likely to connect AAE speakers with adjectives such as “dirty,” “lazy,” and “stupid.” They also linked AAE speakers to less prestigious jobs, or sometimes did not connect them with having a job at all.
The researchers also used a hypothetical criminal defendant scenario. In that setting, the models were more likely to recommend the death penalty for a defendant associated with AAE. The example was constructed, but it showed how language-based bias could become consequential if models are used in high-stakes settings.
Why alignment did not solve the problem
AI companies including OpenAI, Meta, and Google use feedback training to make models less likely to produce harmful responses. In this process, human workers manually shape how a model answers certain prompts. The goal is to move the model toward outputs that better match desired values.
That method appears to work better against explicit stereotypes than against covert ones. According to the paper, GPT-2 was likely to name stereotypes about Black people such as “suspicious,” “radical,” and “aggressive” when prompted directly. GPT-4 no longer gives those associations in the same way.
But the study found that this progress did not remove bias triggered by AAE. The issue is harder than blocking a clearly racist answer to a clearly racist prompt. Dialect prejudice can emerge when the model is not being asked about race at all.
The source article identifies two reasons the problem may have been missed. Companies have been less aware of dialect prejudice as a specific issue, and it is easier to train a model away from overt racist questions than to prevent negative responses to an entire dialect.
Larger models showed stronger covert bias
One of the study’s most important findings was that covert stereotypes grew stronger as model size increased. That creates a warning for companies racing to build larger systems. Bigger models can become more powerful and expressive as training data and parameters increase, but the research suggests size may also amplify hidden racial bias.
This complicates a common assumption in AI development: that more capable models will naturally become better at handling social harms. The source does not show that. Instead, it points to a split outcome: overt racism may be reduced while covert racism becomes harder to see.
The paper was published on arXiv and has not been peer reviewed. Even so, the findings raise an important design question for chatbot makers: whether current tools are enough to identify and reduce bias when it is expressed through language patterns rather than explicit racial categories.
The source article notes that it is not yet clear whether adding more AAE to training data or making feedback efforts more robust will be enough. That uncertainty is central. The research does not present a simple repair; it shows that the current repair strategy may be too narrow.
Why dialect bias matters outside the lab
The study’s examples include extreme scenarios, such as asking a model to judge whether a defendant should receive the death penalty. But the broader concern is not limited to hypothetical tests. AI models are already being considered or used in contexts where language judgments can affect people’s lives.
The source article gives several examples:
- AI-driven translation tools are used when evaluating asylum cases in the US.
- Crime prediction software has been used to judge whether teens should be granted probation.
- Employers using ChatGPT to screen applications might discriminate against candidate names on the basis of race and gender.
- Employers using models to analyze what applicants write on social media could misjudge people who use AAE.
These examples show why covert bias is especially difficult. A model can appear polite, aligned, and non-toxic while still producing judgments that disadvantage a group. If the biased signal is tied to dialect, the harm may be hidden from a user who is only checking whether the model says overtly racist things.
What this means for AI accountability
The study challenges a reactive approach to AI bias. If developers only respond to the latest visible failure, they may reduce one type of harmful output while leaving deeper patterns untouched. Covert stereotypes are harder to detect because they can appear as ordinary evaluations of writing, professionalism, risk, or credibility.
For companies building LLMs, the key lesson is that refusing to produce explicit racist content is not the same as eliminating racial bias. A model can pass one kind of safety test and still fail another. Dialect-based testing may need to become part of how models are evaluated before they are used in sensitive workflows.
For users, the practical lesson is caution. Models should not be treated as neutral judges of people’s character, employability, credibility, or legal outcomes. The research suggests that even advanced systems can attach harmful assumptions to the way someone writes, without ever being told that person’s race.
The deeper issue is not simply whether one model gives one bad answer. It is whether the methods used to improve LLMs are measuring the harms that matter. This study suggests that current feedback training can make racism less visible without making it disappear.