A robot asked to fetch a defibrillator could search in the wrong place if it had stored the device as a fire extinguisher. A new system from MIT researchers is designed to catch and correct that kind of mistaken memory as a robot moves through its surroundings.
When a robot’s first impression is wrong
The system, called UQ-DAAAM, builds on DAAAM, a robot memory framework that uses a vision-language model (VLM) to describe objects seen by onboard cameras. The descriptions are stored in a 3D map of the environment, where the robot can retrieve them later to answer questions.
Those records can be unreliable when an object is only partly visible, far away, blurry, or poorly lit. The earlier system might mistake a gray chimney for a squirrel on a rooftop. If that incorrect label stays in memory, it can affect later tasks that depend on the robot knowing what it saw.
UQ-DAAAM adds a way to estimate how uncertain the robot should be about each description. It considers both the VLM’s confidence in its chosen words and whether descriptions of the same object agree across images taken from different viewpoints.
Using new views to improve the record
As the robot travels, it collects images that may show the same object from several angles. When the system detects a description with too much uncertainty, it selects additional views that could help resolve the doubt.
The selection uses a simple rule, or heuristic, to find useful frames. For the chimney example, a later image showing smoke coming from it could help distinguish the structure from an animal. The researchers mathematically proved that the heuristic can identify useful frames without running the VLM again, reducing the computation required.
After selecting additional captions, the system uses a large language model (LLM) to combine them into a revised description. It gives more weight to newer captions and to descriptions with less ambiguity, then saves the refined record to memory.
What the results show
In the researchers’ evaluation, the refinement step resolved ambiguities for about 92 percent of objects in the robot’s environment while using two additional captions per object. Object description accuracy rose by 5 percent compared to the state of the art.
The system also improved answers to questions about which objects the robot had seen, where it saw them, and when it last encountered them, compared with baseline methods. Those details matter because a robot’s memory is useful only when it can retrieve information that is accurate enough to guide its next step.
“Allowing a robot to know what it doesn’t know is going to be essential for this technology to be deployed safely and have a positive impact on society,” says Luca Carlone, the paper’s senior author. Carlone is an associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), a principal investigator in the Laboratory for Information and Decision Systems (LIDS), and director of the MIT SPARK Laboratory.
From better recall to future tasks
The researchers see potential applications in complex, multistep work, including clearing a dining room table or helping people assemble machinery. Better object descriptions could help a robot keep track of what is around it as it moves between steps.
The next work includes applying UQ-DAAAM to other kinds of robots, such as mobile robotic manipulators, and extending the uncertainty methods to other domains. The team is also studying whether a robot could use uncertainty estimates to identify places it should revisit, along with systems that predict future states of an environment.
The research will be presented at the Conference on Neural Information Processing Systems. It was supported in part by the U.S. Army Research Laboratory and the U.S. Office of Naval Research. The article notes that the work was performed at MIT and is not associated with Amazon.