Covariant is trying to close one of robotics’ most stubborn gaps: helping machines learn useful physical tasks from data, prompts, and feedback instead of relying on narrow, hand-built instructions. Its new model, RFM-1, is designed to give warehouse robots a more flexible way to understand objects, requests, and their own actions.
Why RFM-1 Matters
In the summer of 2021, OpenAI shut down its robotics team and said progress was being limited by the lack of data needed to train robots to move and reason with artificial intelligence. Covariant, a startup spun off in 2017 by three of OpenAI’s early research scientists, says it has made progress on that data problem.
The company’s new model, called RFM-1, combines large-language-model-style reasoning with the physical skills of an advanced robot. It was trained on years of data from Covariant’s fleet of item-picking robots, which customers including Crate & Barrel and Bonprix use in warehouses around the world. It was also trained on words and videos from the internet.
That mix is central to the claim. Covariant is not presenting RFM-1 as just another robot arm controller. It is positioning the system as a more general robotics model that can interpret different kinds of input, reason about a task, and act in a physical environment.
In the coming months, Covariant plans to release the model to its customers. The company hopes RFM-1 will become more capable and efficient as it is used in real warehouse settings.
What The Robot Can Understand
During a demonstration, Covariant cofounders Peter Chen and Pieter Abbeel showed that users can prompt the model with five types of input: text, images, video, robot instructions, and measurements. That multimodal design is important because warehouses are not text-only environments. They are full of bins, products, movement, changing layouts, and practical constraints.
One example involved a bin filled with sports equipment. A user could show the system an image and ask it to pick up the pack of tennis balls. The robot could then grab the item, generate an image of what the bin would look like after the tennis balls were removed, or create a video showing a bird’s-eye view of how the robot would perform the task.
The system can also respond when it anticipates a problem. If it predicts that it cannot grip an object well, it might type back, “I can’t get a good grip. Do you have any tips?” A human could then suggest a change, such as using a specific number of suction cups on the robot’s arms, including eight versus six.
That kind of exchange points to a different relationship between human workers and industrial robots. Instead of requiring complex task-specific code for each situation, the model is meant to adapt through training data and plain-language interaction.
From Fixed Automation To Flexible Instructions
Chen described the system as a step forward for robots that can adapt to their surroundings through training data rather than the complex, task-specific code used by earlier industrial machines. That shift matters because warehouse tasks can vary quickly. A robot may need to deal with different objects, unfamiliar arrangements, and new instructions.
The larger ambition is a worksite where managers can give directions in human language while the robot handles the physical details. The source example was: “Pack 600 meal-prep kits for red pepper pasta using the following recipe. Take no breaks!”
RFM-1 is part of a wider robotics trend. Instead of manually teaching robots how the world works with physics equations and code, researchers are increasingly using large sets of observations. In that approach, a robot learns patterns from examples in a way that is closer to how humans learn from repeated experience.
Chen said the result “really can act as a very effective flexible brain to solve arbitrary robot tasks.” The claim is ambitious, but the demonstration also showed why real-world deployment remains the hard part.
The Warehouse Floor Is The Real Test
Lerrel Pinto, who runs the general-purpose robotics and AI lab at New York University and has no ties to Covariant, said that roboticists have built basic multimodal robots before and used them in labs. What stands out is deploying such a system at scale while allowing it to communicate across so many modes.
For Covariant, the next challenge is data. Pinto said the groups that train strong models will be those with access to large amounts of robot data or the ability to generate those data. Warehouses and loading docks will test whether RFM-1 can handle new instructions, people, objects, and environments.
The model also has clear limitations. During the demonstration, a request to “return the banana to Tote Two” caused the robot to struggle with retracing its steps. It picked up a sponge, then an apple, then other items before finally completing the banana task.
Chen explained the failure directly: “It doesn’t understand the new concept,” adding that it may not perform well in places without strong training data. That matters because human-like learning is only useful if the robot can generalize beyond what it has already seen.
Competition And Unanswered Questions
Covariant is moving into a robotics field that is likely to become more crowded. Earlier this month, Figure AI announced a partnership with OpenAI and raised $675 million from Nvidia and Microsoft. Marc Raibert, the founder of Boston Dynamics, also recently started an initiative to better integrate AI into robotics.
The broader implication is that progress in machine learning may begin to translate more directly into progress in robotics. But the source also raises unresolved questions. If language models are trained on millions of words without compensating authors, robotics models may face similar expectations around training on videos without paying creators.
There are also questions about what the robotics equivalent of hallucinations and bias might look like. In language systems, errors may appear as false or distorted text. In robotics, mistakes can surface through physical actions in real environments.
Covariant’s researchers want RFM-1 to keep learning and refining. Eventually, they aim for the robot to train on videos the model creates itself. Abbeel said, “Training on that will be a reality,” and added, “If we talk again a half year from now, that’s what we’ll be talking about.”
For now, RFM-1 is best understood as both a technical milestone and an open question. It shows how AI robotics may move beyond rigid automation, but its value will depend on how well it performs when warehouse work becomes unpredictable.