Why Multimodal AI Must Learn to Listen and Look

MIT and IBM researchers are studying where multimodal AI models fail to connect language with what they see and hear. Their work explores ways to improve those skills with limited annotated data, including tests that reveal when a model is guessing from a question instead of attending to its audio or video.

WTF Index NEUTRAL
◄ Terminator 1 Idiocracy 1 ►

The story describes research to improve multimodal AI perception, without a clear lean toward harm or human dependency.

Why Multimodal AI Must Learn to Listen and Look

AI models can appear fluent across text, sound, and images while still missing the connection between them. A research team from MIT and IBM is studying those gaps and ways to help models learn from fewer labeled examples.

Connecting what a model hears, sees, and reads

People routinely link sensory information with language. A person may connect a bird’s call with the word “robin,” or recognize and follow a friend named “Sarah” in a crowd. These abilities draw on knowledge across audio, video, and spoken or written language.

Multimodal foundation models are designed to work across different kinds of information, but apparent fluency does not guarantee that they understand the input. The MIT-IBM team, co-led by Jim Glass, a senior research scientist in MIT’s Computer Science and Artificial Intelligence Laboratory, has focused on identifying failures and finding ways to address them.

The team’s approach asks how perception and language can be connected without relying on a large collection of annotated training data. A model may have formed an internal representation of something during training, then lose access to that capability after later fine-tuning. Finding ways to bring such capabilities back is one part of the research.

Learning relationships from matching examples

The project began by teaching models to associate sounds and images. Researchers showed a model examples marked as matching or mismatching, such as an audio caption paired with a picture. The goal was for it to learn that a sound corresponding to “lighthouse” matched an image of a lighthouse, without using text to make that connection.

The work later expanded from speech to general audio, and from still images to video. Models were trained on matching video and audio sequences, as well as mismatched examples. Researchers also masked parts of the image or audio and asked the model to reconstruct them.

These methods let models learn connections beyond spoken words. The team describes models learning to distinguish sounds such as parrots, electric guitars, and acoustic guitars. As large language models emerged, the researchers also began studying how audio could be connected to language, and how available multimodal models could be improved.

Testing whether models use the evidence

Some of the team’s recent work examines whether a model truly uses the visual or audio information in a task. In IPLoc research, the researchers looked at models’ difficulty localizing specific objects from a few examples. Their technique substituted a pseudonym for an object name: for example, replacing “airplane” with “Elizabeth.” That change was intended to make the model rely more on the visual context instead of falling back on what it already associates with the familiar word.

In audio research called OmniR1, the team examined models that listen to audio and describe it. One concern was that a multiple-choice benchmark could reward a model for reading the answer choices rather than listening to the recording. If a question asks which animal is roaring and offers choices such as a chipmunk, a lion, an alligator, and a chickadee, a model may select “lion” based on the options alone.

The researchers found that some models performed well even when given random noise. That result suggested they were responding to the question rather than the sound. Multiple-choice tests are convenient to score, while open-ended answers can require human evaluation or another language model as a judge. But the team’s findings show why evaluation across different types of benchmarks matters: one test may make it easier to mistake a shortcut for real audio understanding.

Adapting models with limited resources

Building a very large foundation model requires resources the research team does not have. Instead, the researchers work with publicly available models and try to adapt them through weakly supervised methods that need less data. Test-time reinforcement learning is one example mentioned in the interview.

The project does not follow a single method or focus on one task. Its researchers pursue different approaches to model limitations, with shared attention to multimodal learning and ways to reduce dependence on extensive annotation. The team has also released public benchmarks, giving other researchers tools to evaluate model performance.

Across roughly two dozen significant collaborative papers, the work has moved from basic links between sound and images toward questions about how language models use audio and visual evidence. The continuing challenge is to make systems that do more than produce plausible answers: they need to connect those answers to what they were actually shown or asked to hear.