Inside a warehouse in San Leandro, California, Encord is testing a new way to build training data for robots. One experiment starts with a simple Jenga game, but the purpose is far from simple: capture what a human sees, does, and may be mentally processing while performing a physical task.
The company believes physical AI faces a data problem as much as a model problem. Instead of only organizing data for robotics companies, Encord is trying to manufacture the kinds of real-world data those companies do not already have.
Why robot training data is becoming the constraint
Encord builds data tooling used to train AI models. Its original role was to help teams working on machine-vision applications annotate data and evaluate models. As robotics customers moved toward end-to-end learning for manipulation tasks, Encord saw a gap: the physical training data needed for those systems was not already available at the required quality or scale.
Vineeth Velmurugan, Encord's head of robot learning, described the problem directly: "The data simply does not exist." He joined Encord after working at OpenAI's robot lab and Berkshire Grey, the warehouse automation firm, and now leads the company's internal data-creation team.
The comparison with large language models is useful but limited. LLMs could draw on enormous amounts of text from the internet and beyond. Robots need examples of how objects move, how hands manipulate tools, how force and precision interact, and how a task unfolds in the physical world. That material is harder to gather and harder to make useful.
Video can help, but Encord's view is that ordinary video does not always provide enough fidelity. Self-driving car companies collect real-world data themselves, but scaling that approach is difficult. Velmurugan says the breakthrough may require a data set something like five times the size of YouTube's video corpus.
What Encord is collecting in the warehouse
Robotics companies are turning to two broad sources of training data. One is egocentric video, collected from workers wearing cameras, often with extra camera angles and other measurements. The other is data generated by robots operated remotely.
Encord works with both. It gathers egocentric data from several factories around the globe, while its San Leandro facility is used to test new data types and produce data sets around specific skills for fine-tuning.
During TechCrunch's visit, pilots - Encord's term for robotic trainers - were using leader-follower rigs. These pair two robotic arms: one is directly controlled by a human operator, while the other mimics the motion. The tasks included pouring coffee from a pot into mugs and stacking poker chips.
The warehouse also held objects used to train manipulators for household-style tasks, including fake flowers in vases, books, plastic vegetables, kitty litter trays and scoops, bags, and bundles of wires. These objects matter because physical AI systems need to learn from the ordinary messiness of the real world, not just from clean demonstrations.
One station focused on a task that shows why precision remains difficult. Sofia Infante, another pilot, controlled robotic arms to plug and unplug ethernet cables from the back of a server. Data center operators would like work like that to be automated, but the source article makes clear why it remains hard: pincers are less dextrous than human fingers and do not match the degrees of freedom humans rely on in their arms.
Where brain waves fit into physical AI
The Jenga experiment adds another layer to the data. Andrew Ceja, a pilot at Encord, wears a headset with a camera that tracks what he sees. That part is already common in robot data collection. What makes this setup different is that the headset also includes sensors measuring his brain waves while he removes blocks from the tower.
The headset was built by Zander Labs, a German neuroscience startup. Zander Labs is exploring whether brain activity can help identify mental states such as error, intent, and surprise. The premise is that those signals could create a more useful data set for training models.
Encord's work with Zander Labs is still a trial run. The company says the goal is to create an initial brain wave-tagged data set, run it through customer robotics models, and then judge whether it improves performance before deciding whether to expand the effort.
Lucas Gehrke, a Zander neuroscientist supervising the work, says the amount of brain activity used during a task may offer model builders clues about when to use their highest-effort models. In other words, the signal may help distinguish moments that are routine from moments that demand more computation or care.
Velmurugan calls this the "bleeding edge" of work on the robotics data bottleneck. It is not presented as a proven solution yet. It is an experiment aimed at finding out whether mental-state signals can make physical AI training data more valuable.
Beyond video: muscles, annotations, and cost
Brain waves are not the only new data modality Encord is exploring. The company is also developing a system that uses sensors strapped to the forearm to detect electrical signals in muscles. The goal is to infer a 3D depiction of where the hand is at any moment.
That matters because video of human hands manipulating objects often does not capture the entire hand. If arm sensors can fill in what the camera misses, models may receive a more complete representation of the action.
Encord also annotates its data sets with physical descriptions of what each video contains. One example from the source is "right hand tightens bolt." These labels are meant to help LLM-based models understand what is happening in the scene.
Velmurugan estimates that this kind of dense annotation is worth 100 times as much as "junky ego data" for training specific tasks, while costing 20 times more to produce. On paper, he frames that as a good trade. The tension is that 20 times more is still expensive.
That cost is the dividing line between physical AI and the large-language-model playbook. Text could be scraped from the web at low cost for frontier labs. Physical training data has to be created, staged, recorded, cleaned, and annotated. That makes data generation a business in its own right, not just a research support function.
The workforce behind the training data
Encord's facility has around a dozen pilots working on these tasks. Infante and Ceja are part of a growing workforce building the raw material for neural networks. Both previously worked at Scale, another AI data annotation firm, before joining Encord.
Ceja previously worked at a waste management company, where his interest in technology led him to maintain a robotic trash sorter. At Encord, he now works through varied training challenges for robots. After the Jenga tower falls, he describes the job this way: "It's something new every day!"
The broader point is clear: physical AI depends on more than better algorithms. It also depends on people, tools, sensors, and carefully designed tasks that can translate human physical skill into machine-readable training data.