Teaching a robot a new task can mean hours of repeated demonstrations. Toyota Research Institute (TRI) is developing a method that uses those demonstrations to train robot skills overnight, with the aim of helping machines cope with tasks and surroundings that change.
From demonstrations to overnight training
TRI’s approach starts with teleoperation: a person remotely controls a robot to show it how to perform a behavior. Researchers repeat the task several dozen times, a process that typically takes about an hour for a basic behavior. The robot can then use the collected examples to train its neural networks overnight.
The system combines traditional robot learning methods with diffusion models. In this context, diffusion policy uses a process that removes noise from randomized images to help generate robot actions. The idea draws on methods also used in generative AI, but here the goal is to produce behaviors for a physical machine.
TRI says it has trained robots on 60 skills using the method. That count signals progress, though the research team says that building a robot capable of learning broadly remains a longer-term challenge.
Why touch matters for physical tasks
The demonstrations can include information from both sight and force feedback. A teleoperation device can transmit force between the robot and the person controlling it, allowing the person to feel how the robot interacts with objects. TRI researchers say this can help coordinate actions that are hard to demonstrate through remote control alone.
Combining senses gives a robot more clues about what is happening. For example, force feedback can help it recognize whether it is holding a tool correctly. When training data links what the robot sees with what it feels, the system can use its built-in sensors to reproduce the demonstrated activity.
TRI’s early tactile experiments showed different results across tasks. Robots flipped pancakes successfully in 27 out of 30 attempts, a 90% success rate, compared with 83% in trials without tactile sensing. For dough rolling and food serving, reported success rates with tactile sensing were 96% and 90%. Without it, those figures fell to 0 and 10%, respectively.
Adapting beyond predictable settings
Robots often work more easily in carefully structured environments than in places where objects and routines vary. A warehouse, for example, may be arranged consistently, even as robots navigate around moving people or forklifts. Homes are less predictable: furniture moves, messes appear, and objects do not always return to the same place.
TRI has focused on systems that could help older people live independently. For that goal, a robot would need to work in different environments and respond to changes within them. Programming every possible exception in advance is a brute-force approach, and the range of everyday variations makes it difficult to rely on that alone.
Researchers describe adaptable, multi-task systems as a move away from robots trained to repeat one job. Still, they caution that robots that can credibly be called general purpose are not here yet. The current work aims to fill in a gap left by large language models: those models may help a robot understand a high-level instruction, but tasks such as plugging in a USB device or picking up a tissue require learned physical behavior.
Building toward a larger library of behaviors
TRI’s next ambition is to expand the set of skills robots can learn and explore how those skills combine. Vice President of Robotics Research Russ Tedrake said the team had trained 60 skills already, with targets of 100 skills by the end of the year and thousands of skills by the end of next year. The team is studying how many skills are needed before robots can handle something genuinely new.
A larger library could let robots link smaller behaviors together and use them in unfamiliar situations. The research also points toward fleet learning, in which a cloud-based system lets robots learn from one another’s experiences. Both ideas depend on continued progress in training and adapting robot skills.
For now, overnight learning is a research method, not evidence that robots can handle any household task on their own. TRI’s experiments show how demonstrations, sensory information, and training methods can work together to improve specific physical skills. The broader question is how far that combination can scale.