Deep-learning systems can be difficult to understand from the outside. Their internal networks contain many processing nodes, and figuring out what those nodes represent can help researchers anticipate how a model may behave when put to use.
Jacob Andreas, an associate professor in MIT’s Department of Electrical Engineering and Computer Science and a member of the Computer Science and Artificial Intelligence Laboratory, is pursuing a more direct route: training models to explain aspects of their own operation in human language.
Turning internal computations into descriptions
Andreas’s research asks whether language models can help make the inner workings of neural networks more legible. One family of techniques developed by his group trains large language models to describe what is happening inside other neural networks.
These methods can attach a natural-language description to each information-processing node in a large network. A node, for example, might be described as identifying “patches of grass near animals” in images. Such descriptions offer researchers a way to connect complex computations with concepts people can recognize.
The broader aim is to make model behavior easier to study and predict. If researchers can better understand what a trained system has learned, they may be better equipped to judge how it is likely to act in real-world use and whether it fits its intended application.
Why language is central to the work
Andreas’s group also studies how machines learn language and how they might learn from human guidance. The researchers want intelligent systems to learn about the world and communicate effectively with people working alongside them.
That interest links the practical problem of interpreting neural networks with deeper questions about learning and thought. Andreas has said that language-processing problems brought such questions into view for him: they connect how a system represents meaning with how it can communicate that meaning.
His early research included machine translation, using statistical methods for representing meaning as a foundation for systems that could translate passages of text. Later, he worked on AI that combined language with visual information, including systems trained to “see” and understand the world around them.
From statistical methods to deep learning
During Andreas’s graduate studies at the University of California at Berkeley, natural-language processing was shifting from statistical methods toward deep learning. Statistical approaches rely on transparent mathematical formulas to analyze text. Deep-learning systems instead use algorithms that learn complex meanings from large amounts of text data, but their internal processes can be harder to interpret.
Neural networks also offered researchers a shared toolkit for working with language and vision data. Andreas completed a thesis project on neural module networks: small neural-network components that handle tasks such as locating objects or performing logical reasoning within a larger system.
Alongside that work, he explored interpretability and ways to describe a model’s internal states in natural language. Those questions later became part of his group’s research at MIT.
Making models more useful in practice
Andreas joined the MIT faculty in 2019 after spending a year at the startup Semantic Machines, which spun out of his lab at Berkeley. His group’s work now includes studying the algorithms large language models use as they learn and process information, as well as developing tools to help explain other neural networks.
For high-stakes applications, understanding a model’s internal operations matters because researchers need to assess what it has learned and how it might behave. Andreas’s approach seeks to translate complicated neural computation into descriptions that people can examine and discuss.
That translation is not presented as a replacement for studying models. It is a way to make their learned representations more accessible, supporting the larger goal of designing deep-learning systems that are more accurate, robust, and appropriate for the tasks they are meant to perform.