A robot may be told to keep coffee away from a laptop. But a person giving that instruction may also expect it to avoid nearby electronics, without naming each device. A learning platform called FLEX is designed to help robots make those broader connections from natural language commands.
From literal instructions to broader intent
Researchers from MIT and New York University developed FLEX as a feature-editing pipeline for robot motion-planning. The work is to be presented at the annual Conference on Robot Learning this month. It aims to let people guide robots with ordinary conversational language instead of having to specify every detail of a task.
In a simulated coffee-serving scenario, a person types “Don’t spill on the laptop!” as a robot arm carries a cup across a cluttered table. FLEX helps the robot infer why the laptop matters: liquid could damage electronics. It then adjusts its route to avoid a nearby phone too, even though the person did not mention that device.
The system uses a large language model, or LLM, to connect the specific objects named in a command with broader patterns. The researchers used GPT-4o in FLEX’s generalization step. The aim is to turn a one-off correction into a useful preference that can guide the robot’s path.
How FLEX updates a robot’s plan
FLEX works in three steps: “Add,” “Generalize,” and “Mask.” Together, they let the robot respond to what a person says, infer a category of concern, and keep its working list tied to the current surroundings.
- Add: The robot puts an object mentioned in a command onto a list of features that matter for the task. If someone says “Don’t drop it on my mug!” while the robot moves a bowling ball, the mug becomes something to avoid.
- Generalize: The LLM looks for relationships among the objects the person has flagged. In this example, it can infer that fragile objects may need protection, even when the person has not named every fragile item.
- Mask: The robot’s computer vision system checks which listed objects are no longer present and removes them from the list.
The full process takes about five seconds, according to the researchers. The update step addresses a limitation of simply adding every correction to a growing list: as the environment changes, some listed objects may no longer be relevant, and people may tire of naming every item they want the robot to avoid.
What the simulations showed
The researchers tested FLEX in simulations involving a robot arm serving coffee, moving a candle, and carrying a bowling ball. Volunteers gave commands in natural language, including “Don’t spill on my laptop,” “Stay away from the stack of paper,” and “Don’t drop on the mug.”
In these scenarios, the robot followed the named instruction and generalized its caution. It avoided electronics when told not to spill on a laptop, kept away from flammable objects after being warned about paper, and avoided dropping its load on fragile things after the mug instruction.
The examples show how a single sentence can communicate more than an object to steer around. The command can point to a concern, such as damage to electronics or fragile items, that applies to other things in the robot’s path. FLEX is intended to help translate that concern into motion-planning adjustments.
A more natural way to guide robots
Robots use mathematics to plan trajectories, but the researchers say people should not have to learn that math to give instructions. Natural language can make it easier to redirect a robot or describe a task. FLEX extends this approach by using what a person says to infer an unstated preference, rather than treating every command only as a direct physical instruction.
That could matter in homes, where objects and priorities differ from one person to another. A robot cannot assume it already knows which items matter to a particular household. The researchers’ goal is for it to pick up some of those priorities from a sentence or two, reducing the need for people to spell out every rule.
The reported results are from simulated tasks, so the article describes a demonstrated approach in virtual scenarios. The study was led by Andreea Bobu, assistant professor of aeronautics and astronautics at MIT. Co-authors are Audrey Lee SM ’26 and Ilia Sucholutsky of New York University.