Robots need to do more than move objects in a quiet, empty room. They may need to sort items, help tidy a living room, or move alongside a person without getting in the way. Research from Nvidia and Meta uses simulated worlds to let AI agents practice these kinds of physical tasks at a much faster pace than repeated attempts in the real world would allow.
Nvidia uses an AI model to help design the lesson
Teaching an agent to control a robot can involve repeating a task such as opening a drawer and putting something inside hundreds or thousands of times. Doing that on physical equipment can take days. In a realistic simulation, an agent may learn to perform nearly as well in a minute or two.
Nvidia’s EUREKA system adds automation to this approach. It uses a large language model to help write reinforcement learning code, which guides an AI agent as it improves at a task. The system can also revise its own code as it goes.
A task such as picking up objects and sorting them by color can be described in different ways. The reward function—the rules that tell an agent which outcomes count as better—might favor fewer movements or a shorter completion time. Choosing an effective version can take trial and error. Nvidia’s researchers found that a code-trained language model often produced more effective reward functions than humans did, and could adapt them to other applications.
The pen-spinning example from the research was performed in simulation. Nvidia also reported strong results on virtual dexterity and locomotion tasks, including using scissors. The work reduced the human time and expertise needed to create those simulated skills, but getting an action to work on a real robot remains a separate challenge.
Meta puts people into the robot’s practice space
Meta’s Habitat project develops simulated environments for AI agents to navigate and interact with. Its first version appeared in 2019, with carefully annotated, nearly photorealistic 3D spaces. Habitat 2.0 added more environments and made them more interactive and physically realistic, alongside a growing library of virtual objects.
Habitat 3.0 adds human avatars that can share a simulated space through virtual reality. This lets a person, or an agent modeled on human behavior, interact with a robot and its surroundings while the robot practices.
That matters for household tasks because people do not stay still while a robot works. A robot asked to carry dishes from a coffee table to the kitchen and put clothing in a hamper could develop a plan that fails as soon as someone walks through the room—or a person might do part of the work. Repeated practice with a human or human-like agent in the same space gives the robot a chance to learn how to work alongside them.
Practice includes cooperation and movement around people
Meta calls the cleanup task “social rearrangement.” The idea is to move objects around a shared space while accounting for another participant’s actions. The system also studies “social navigation,” in which a robot follows someone without being obtrusive. The source gives the example of a small robot accompanying someone in a hospital on a walk to the bathroom, so it can stay within audible range or watch them for safety reasons.
These examples show why physical skill alone is not enough. A robot may be able to carry an object, but in a shared space it also needs to respond to where a person is and what they are doing. Simulation offers a way to repeat those interactions quickly and explore how the robot behaves as the situation changes.
More realistic scenes and shared tools
Meta’s HSSD-200 database provides 3D interiors with improved fidelity. The researchers found that training in around a hundred high-fidelity scenes produced better results than training in 10,000 lower-fidelity ones. That finding points to a tradeoff researchers can consider when building practice environments: a smaller set of more realistic spaces may be more useful than a much larger set of less detailed ones.
Meta also introduced HomeRobot, a robotics simulation stack for Boston Dynamics’ Spot and Hello Robot’s Stretch. Its goal is to standardize basic navigation and manipulation software so researchers can spend more time on higher-level work.
Habitat and HomeRobot are available under an MIT license on their GitHub pages. HSSD-200 is available under a Creative Commons non-commercial license. Together, the projects offer researchers tools for practicing navigation, object handling, and shared-space behavior in simulation. They do not erase the challenge of moving from a virtual environment to the physical world, but they provide a faster setting for developing and studying those capabilities.