A digitized bookstore could be searched for a particular title, or a kitchen scene could point to the paper towel needed to clean up a spill. LERF connects natural-language queries with 3D environments built using Neural Radiance Fields, giving users a way to find relevant objects and places inside a scene.
Turning a 3D scene into something searchable
Neural Radiance Fields, or NeRFs, can turn real places into detailed 3D representations. LERF, short for Language Embedded Radiance Fields, adds language-based search to those environments by integrating CLIP vectors into a NeRF scene.
The system produces 3D relevance maps: regions of a scene that match a query. A person can search those maps with natural language rather than having to identify an object in advance through a separate training process.
In a bookstore example, a user could enter a specific book title. LERF can locate and tag that book with pixel accuracy on the first try, in what the researchers describe as zero-shot recognition. Their method does not require region proposals, masks, or fine tuning.
Queries can describe more than an object’s name
Language searches can refer to an object’s color, shape, function, name, or brand. The system can also distinguish between different kinds of donuts, including chocolate and blueberry. That flexibility makes the scene useful to explore with descriptions that go beyond a fixed list of object labels.
The researchers also tested how LERF could connect language-model suggestions to the objects and areas in a 3D scene. They used ChatGPT to generate steps for cleaning up a kitchen after coffee had spilled. LERF mapped the suggested actions to relevant parts of a kitchen NeRF.
For one step, the system marked a paper towel over the sink as relevant to wiping up the coffee. The spill itself also appeared in the scene, but it received less relevance for a search about the paper towel. In this example, the map links a suggested action to the object that could help carry it out.
Potential uses—and a limit of static scenes
The approach could make digitized real-world places easier to search. The source article points to work on integrating NeRFs of places such as restaurants and stores into Google Maps. With LERF, someone could search a scanned location virtually for objects or features described in ordinary language.
There is an important constraint: LERF scenes are static. That makes the system less suitable for a real-time question such as finding the nearest supermarket. The article says multimodal search using ordinary 2D webcam images would be more appropriate for that kind of query.
A guided virtual reality tour of a real store is a different use case. A NeRF can represent the store as a 3D environment, while LERF can help a visitor search within that environment. The combination could let someone look for a particular product or feature during a virtual visit.
Meta is also researching NeRFs, with the aim of letting people bring real objects into digital worlds using smartphone scans. LERF is part of a broader effort to connect language-based systems with digital places that can closely resemble the real world.
From virtual search to robotics research
The research team sees possible applications in robotics, including visual robot training in simulations, studying the capabilities of visual-language models, and interacting with 3D worlds. A natural-language map of a scene could help connect a task description with the objects and locations that matter to it.
The team plans to integrate LERF into the open source NeRF software "Nerfstudio." The work presents a way to search within a 3D scene using language, while the static nature of those scenes sets a clear boundary on where the approach fits.