A neuron in an image model appeared to respond to both cats and cars, a finding once taken as evidence that neural networks jumble unrelated concepts together. Nick Cammarata now sees a shared visual pattern behind that response—and says researchers may need to look more patiently at what models have learned. His argument has sparked optimism about understanding AI, alongside calls for stronger evidence.
Why the old picture was hard to untangle
Neural networks are the software behind most AI models. For years, researchers have described their internal workings as tangled and difficult to interpret. The models are often compared to gardens that grow, rather than bridges built to a clear plan.
Cammarata helped develop methods for examining models while working at OpenAI. This kind of work, known as mechanistic interpretability, includes probing individual neurons to learn what they respond to. Researchers hoped such observations would reveal how a model represents information.
In 2020, Cammarata and colleagues examined Inception v1, a small image-recognition model. One neuron appeared to activate for images of both cars and cats. They interpreted this as evidence that a neuron could handle multiple concepts at once, a phenomenon called superposition.
That result helped shape a broader expectation: looking inside a model would reveal a knot of overlapping meanings. Researchers have since tried to separate those meanings, but progress has been slow. Modern language models are so large that researchers can explain only small parts of them at a time.
A shared pattern in cats and cars
At Goodfire, Cammarata has spent much of the last year examining neurons, using tools he built and teams of AI agents to help interpret what he sees. He says the more closely he looked, the more often a neuron’s responses seemed to make sense. Sometimes, he says, he could map out whole circuits.
He returned to the car-and-cat example with a different question. Instead of treating the images as unrelated concepts packed into one neuron, he looked for a pattern they might share. He noticed that both can contain a prominent V shape: a cat’s face or the front of a car viewed from the side.
In follow-up experiments, the neuron also responded to the corner of a pool table, the pointed nose of a Japanese high-speed train, and a fire boat shooting water into the air from its deck. Images with a round O shape could turn it off. Cammarata’s interpretation is that the neuron had picked up on a visual feature that appears across different kinds of images.
The images and model had not changed, he says. What changed was the way he approached the evidence: he expected the network to define patterns and looked for them. That experience led him to reconsider how much of a neural network might be legible if researchers do not give up too quickly.
What the claim does—and does not—show
Cammarata is not claiming to understand language models in full. He says, however, that he no longer believes a complete understanding is impossible. In his view, automated scientists could eventually help people work through models much faster than a person examining one circuit at a time.
That possibility remains a proposal, not a demonstrated result. Inception v1, built by Google in 2014, is far simpler than the leading models of today. Cammarata and colleagues have begun studying modern language models and plan to publish technical results, but the source article reports that those details are not yet available.
Researchers who question the excitement say that identifying what individual neurons do is only part of the task. Understanding how neurons connect into circuits matters too. Stanford University’s Chris Potts says that a large number of examples is not a substitute for rigorous evaluation, while Jacob Hilton and Neel Nanda also caution against drawing broad conclusions from the current work.
Cammarata acknowledges that some readers have taken his argument too far. Open questions remain, including how much model behavior comes from combinations of neurons rather than individual neurons. His claim is that understanding may be achievable, not that interpretability has been solved.
Why the debate matters for AI safety
The argument has landed as researchers and policymakers are concerned about controlling powerful AI systems. The source article describes recent worries about agents breaking containment and causing problems on the public internet without companies knowing what they were doing. Better ways to interpret models could help researchers investigate why they behave as they do.
For Cammarata, the prospect of finding structure inside neural networks offers a reason for hope. For skeptics, the next step is evidence that the approach works on modern language models and explains how their internal parts fit together. The discussion turns on that distinction: a promising way to look at neurons can motivate research, while the larger question of understanding and controlling advanced models remains open.