AI is changing robotics, but not in the same way it changed chatbots. Robots need to understand the physical world, and that requires data that is harder to collect, more expensive to label, and more legally complicated to use.
The result is a new race inside the robot race: companies and researchers are competing to find useful training data before their rivals do. That competition is shaping which robots become more capable, where they can work, and how quickly they move from controlled settings into homes, warehouses, hospitals, and other real-world environments.
Why robots need different data
Since ChatGPT was released, many people have become used to interacting with AI tools directly. Robots remain much less common in daily life. For many people, the most advanced robot they encounter is still a vacuum cleaner, unless they work in logistics or undergo complex surgery.
Roboticists believe that new AI techniques could change that. The goal is to build machines that can move through unfamiliar environments and handle tasks they have not seen before. Russ Tedrake, vice president of robotics research at the Toyota Research Institute, describes the current pace of the field this way: “It’s like being strapped to the front of a rocket.”
But robots cannot be trained only on the same kind of material used for advanced AI models, such as text, images, and videos scraped from the internet. They must learn how objects move, how tools behave, and how actions unfold in three-dimensional space.
Simulation can help, but it has a major weakness. Robots trained in simulated environments can still fail when moved into the real world, a problem known as the “sim-to-real gap.” That keeps real-world data at the center of robotics progress.
The premium data comes from demonstrations
For decades, many robots were trained for specific tasks, such as picking up a tennis ball or doing a somersault. Humans learn through observation and trial and error, but many robots relied on equations and code. That made progress slow and limited the ability to transfer one skill to another task.
Now researchers are shifting toward data-driven learning. A robot can be shown hundreds of demonstrations of a task, such as washing ketchup off a plate using robotic grippers, and then learn to imitate the task without being explicitly taught what ketchup is or how a faucet works.
At the Toyota Research Institute, researchers gather this kind of demonstration data through teleoperation. A human can guide a robotic arm through the same task many times. One example is flipping a pancake 300 times in an afternoon.
This method creates valuable training data because the robot gets repeated examples of the same physical process. TRI says the model can process the data overnight, and the robot can often perform the task autonomously the next morning.
The limitation is cost and scale. Teleoperation takes time, and it depends on access to expensive robots. Shuran Song, head of the Robotics and Embodied AI Lab at Stanford University, has worked on a lower-cost alternative: a lightweight plastic gripper that can collect data while a person does everyday activities such as cracking an egg or setting the table.
Open data helps, but private fleets have an edge
Some robotics researchers are trying to solve the scarcity problem by sharing data. The Distributed Robot Interaction Dataset, or DROID, was created by researchers at 13 institutions, including Google DeepMind, Stanford, and Carnegie Mellon.
DROID contains 350 hours of data from people performing tasks such as closing a waffle maker and cleaning up a desk. Because the data was collected with hardware that is common in robotics, researchers can use it to build AI models and test those models on equipment they already have.
Another effort, the Open X-Embodiment Collaboration from Google DeepMind, aggregated data on 527 skills collected from different types of hardware. That data helped build Google DeepMind’s RT-X model, which can translate text instructions such as “Move the apple to the left of the soda can” into physical movements.
Open-source robotics data can produce impressive results, but it has limits. Lerrel Pinto, who runs the General-purpose Robotics and AI Lab at New York University, says the available open-source data is not enough to match the proprietary models being built by leading private companies. “The biggest limitation is the data,” he says.
Private companies with deployed robots have a powerful advantage: their machines keep collecting data during normal work. Covariant, founded in 2017 by OpenAI researchers, deploys robots that identify and pick items in warehouses for companies including Crate & Barrel and Bonprix.
When a Covariant robot fails to pick up a bottle of shampoo, that failure becomes a useful training example. The company uses this flow of data to improve its systems. Earlier this year, Covariant released a foundation model called RFM-1, allowing customers to interact with commercial robots in ways that resemble chatbot conversations.
Video offers scale with trade-offs
Because teleoperation data is scarce, some researchers are looking at video. Videos are easier to produce, but they usually lack kinematic data, which tracks the exact movement of a robotic arm through space.
Researchers from the University of Washington and Nvidia have built a mobile app that uses augmented reality to help bridge that gap. Users record themselves doing simple tasks with their hands, such as picking up a mug, and the AR program translates the action into waypoints for robotics software.
Meta AI is also working with video at scale through Ego4D. The dataset includes more than 3,700 hours of video taken by people around the world doing activities such as laying bricks, playing basketball, and kneading bread dough. It is divided by task and includes thousands of annotations describing what is happening in each scene.
YouTube offers an even larger pool of video, but much of it is not directly useful for robotics. A video may show a person throwing a Frisbee, but it usually does not explain the acceleration, rotation, or physical dynamics in the way a robot would need.
Still, researchers are trying. Emmett Goodman trained an AI model on thousands of hours of open-surgery videos gathered from YouTube, with identifiable information removed. The work, described in a paper in JAMA in December 2023, helped identify segments of operations from video and pointed toward future training data, though patient privacy and informed consent remain barriers at scale.
The legal questions are still unsettled
The search for robot training data is moving into legally uncertain territory. Language-model companies are already facing disputes over credit, copyright, and the use of online material in training sets.
The New York Times has filed a lawsuit alleging that ChatGPT copies the expressive style of its stories when generating text. OpenAI’s video generation tool Sora also drew scrutiny after the company’s chief technical officer said it was trained on publicly available data, and YouTube’s CEO argued that training on YouTube videos would violate the platform’s terms of service.
Frank Pasquale, a professor at Cornell Law School, says this is “an area where there’s a substantial amount of legal uncertainty.” If robotics companies use copyrighted works in training sets, it is unclear whether that use fits within fair use. Pasquale points to the 2015 Google Books case as a precedent that may tilt slightly in tech companies’ favor.
For now, the core issue is simple: better robots need better data. The companies and labs that can gather, label, share, or legally defend the largest and most useful datasets will have a major influence on what the next generation of robots can actually do.