Waabi is bringing generative AI deeper into autonomous driving with Copilot4D, a model built to anticipate how traffic around a vehicle may change in the near future. Instead of generating pictures from text, the system works with lidar data and produces a future lidar representation of a driving scene.
The announcement matters because self-driving systems depend on more than seeing what is nearby. They also need to reason about what pedestrians, trucks, bicyclists, and other vehicles may do next.
What Copilot4D is designed to predict
Copilot4D was trained on large amounts of data from lidar sensors. Lidar uses light to sense how far away objects are, creating point clouds that form a 3D map of a vehicle’s surroundings.
Waabi says the model can be prompted with a situation, such as a driver recklessly merging onto a highway at high speed. From that setup, Copilot4D predicts how nearby vehicles will move and generates a lidar view of the scene 5 to 10 seconds into the future.
That future view might show a serious outcome, such as a pileup, if the model estimates that the surrounding traffic would respond in that way. The point is not simply to visualize a scene, but to give autonomous driving software a way to reason about how the road could develop in the next few seconds.
According to Waabi CEO Raquel Urtasun, the announcement covers the initial version of Copilot4D. She says a more advanced and interpretable version is deployed in Waabi’s testing fleet of autonomous trucks in Texas, where it helps the driving software decide how to react.
Why generative AI is entering self-driving
Autonomous driving has long used machine learning for tasks such as route planning and object detection. The newer bet from some companies and researchers is that generative AI can help autonomy move forward by taking in information about a vehicle’s surroundings and generating predictions about what may happen next.
Waabi is not alone in that direction. Wayve, a competitor, released a comparable model last year that is trained on video collected by its vehicles. Waabi, however, is building its generative model around lidar rather than cameras.
The approach resembles how image and video generators such as OpenAI’s DALL-E and Sora process data. Copilot4D breaks lidar point clouds into chunks, similar to how image generators divide photos into pixels. Based on what it learned during training, the model predicts how all points of lidar data will move.
By repeating that process continuously, the system can produce predictions 5-10 seconds into the future. For Waabi, that short window is central: many driving decisions depend on understanding what is likely to happen almost immediately, not far down the road.
The AI-first argument
Waabi is part of a small group of autonomous driving companies, including Wayve and Ghost, that describe themselves as “AI-first.” For Urtasun, that means building systems that learn from data rather than relying on hand-taught reactions to specific driving situations.
The companies using this approach are betting that it could reduce the amount of road-testing needed for self-driving cars. That question has become especially charged following an October 2023 accident in which a Cruise robotaxi dragged a pedestrian in San Francisco.
Waabi’s use of lidar also reflects a specific view about what advanced autonomy requires. Urtasun argues that lidar is necessary for companies pursuing Level 4 autonomy, the level where a vehicle does not require human attention to drive safely.
“If you want to be a Level 4 player, lidar is a must,” says Urtasun.
Her reasoning is that cameras can show what the vehicle sees, but are less capable at measuring distances and understanding the geometry around the car. Lidar is therefore positioned as a key input for a system that needs to anticipate movement in a 3D environment.
What the model will and will not train
Copilot4D can generate videos showing what a car would see through its lidar sensors. But Waabi says those videos will not be used as training material in the company’s driving simulator, which is used to build and test its driving model.
The reason is hallucination risk. If Copilot4D produces a false or distorted future scene, Waabi does not want that generated output taught back into the simulator.
That separation highlights a practical limit of generative AI in autonomous driving. A model may be useful for prediction and decision support, while still being too risky to feed directly into the training loop that shapes the driving system itself.
The open-source question
Bernard Adam Lange, a PhD student at Stanford who has built and researched similar models, says the underlying technology is not new. What stands out to him is seeing a generative lidar model move beyond research and scale toward commercial use.
“It is the scale that is transformative,” he says.
He says models like this could help the “brain” of an autonomous vehicle reason faster and more accurately, with possible use in downstream tasks such as object detection and motion prediction.
Still, the usefulness of Copilot4D depends heavily on how reliable its predictions remain as they look farther ahead. Motion prediction models generally degrade the farther into the future they are asked to project. Urtasun says 5 to 10 seconds is enough for most driving decisions, but Waabi’s highlighted benchmark tests are based on 3-second predictions.
Chris Gerdes, co-director of Stanford’s Center for Automotive Research, says that difference will matter when evaluating how useful the model is for real decisions.
“If the 5-second predictions are solid but the 10-second predictions are just barely usable, there are a number of situations where this would not be sufficient on the road,” he says.
The announcement also raises the familiar generative AI question of whether models should be open-source. Releasing Copilot4D could help academic researchers who lack access to large data sets, allow independent safety evaluation, and potentially move the field forward. It would also give Waabi’s competitors more visibility into its work.
Waabi has published a paper explaining how the model was created, but it has not released the code. Urtasun says she is unsure whether the company will do so.
“We want academia to also have a say in the future of self-driving,” she says, adding that open-source models are more trusted. “But we also need to be a bit careful as we develop our technology so that we don’t unveil everything to our competitors.”