Robots can often learn a skill in one setting, then struggle when the room, object position, or layout changes. A research team from New York University, Meta, and Hello Robot is trying to solve that practical gap with AI models designed for familiar tasks in unfamiliar places.
The models are called robot utility models, or RUMs. They focus on helping a robot do specific things it already knows how to do, but in new surroundings where it has not been specially trained.
What RUMs Are Built To Do
The researchers developed five AI models for five separate tasks: opening doors, opening drawers, picking up tissues, picking up bags, and picking up cylindrical objects. The central idea is not to make one robot do everything. It is to make a robot perform a narrow set of useful tasks more reliably across many different environments.
That distinction matters because robots have typically needed fresh training data when they are moved into unfamiliar settings. Gathering that data can be slow and expensive, because robotic training data has to be collected physically. Unlike large language models, which can be trained on information scraped from the internet, robots need demonstrations from the physical world.
Mahi Shafiullah, a PhD student at New York University who worked on the project, framed the problem this way: “In the past, people have focused a lot on the problem of ‘How do we get robots to do everything?’ but not really asking ‘How do we get robots to do the things that they do know how to do—everywhere?’”
That question shaped the work. Instead of chasing a broad, all-purpose robot, the team asked how a robot could learn something like opening any door, anywhere.
How The Training Data Was Collected
To gather the data behind the models, the researchers used a simple setup: an iPhone attached to a cheap reacher-grabber stick, the kind typically used to pick up trash. The tool was a new version of one used in previous research.
The team used that setup to record around 1,000 demonstrations in 40 different environments for each of the five tasks. The environments included homes in New York City and Jersey City, and some demonstrations had already been gathered as part of previous research.
Those demonstrations were then organized into five data sets. Learning algorithms were trained on those data sets to create the five RUM models.
The approach is important because it gives roboticists a more repeatable way to build task-specific models. If a robot can learn from a focused set of physical demonstrations and then apply the skill in new places, the cost and time needed to prepare robots for homes could fall.
Testing The Models In New Environments
The RUM models were deployed on Stretch, a robot made up of a wheeled unit, a tall pole, and a retractable arm holding an iPhone. The researchers used Stretch to test whether the models could perform their assigned tasks in new environments without additional tweaking.
In those tests, the models reached a completion rate of 74.4%. The team then added another step: images from the iPhone and the robot’s head-mounted camera were given to OpenAI’s recent GPT-4o LLM model, which was asked whether the task had been completed successfully.
If GPT-4o said the task was not complete, the researchers reset the robot and tried again. With that check-and-retry process, the success rate rose to 90%.
The result shows a practical pattern for robot learning. A robot can attempt a task using a utility model, then another AI system can help judge whether the task worked. The source does not show this as a perfect solution, but it does show a way to improve results without retraining the robot for every new room.
Why Real-World Settings Matter
A major challenge for robotics is that lab conditions do not fully represent what happens in real homes and kitchens. A model can appear strong in a controlled space, yet fail when objects, layouts, or surfaces differ from what it has already seen.
Mohit Shridhar, a research scientist specializing in robotic manipulation who was not involved in the work, welcomed the focus on testing in varied homes and kitchens. “It’s nice to see that it’s being evaluated in all these diverse homes and kitchens, because if you can get a robot to work in the wild in a random house, that’s the true goal of robotics,” he says.
That outside view highlights the broader stakes. Robots intended for homes need to handle spaces that are not arranged for them. They need to deal with ordinary variation without requiring a roboticist to rebuild the training process every time.
What This Could Mean For Home Robots
The project points toward a more modular way to teach robots. Instead of training a robot from scratch for every environment, developers could build utility models for specific tasks and make them easier to reuse.
Shafiullah says the project could become a general recipe for building other utility robotics models. That could help people who are not trained roboticists deploy future robots in their homes with less extra work.
He describes the longer-term goal plainly: “The dream that we’re going for is that I could train something, put it on the internet, and you should be able to download and run it on a robot in your home,” he says.
For now, the reported work is limited to five basic tasks. But those tasks are the kind of physical actions that matter if robots are ever going to be useful outside labs: opening, reaching, grasping, and handling everyday objects in places they have not seen before.