AI systems can work with language and 2D images, but those inputs do not by themselves give a model a clear map of a physical environment. A research team has proposed 3D language models, or 3D LLMs, that use three-dimensional data to help connect what a model says with where things are and how a space is arranged.
Giving language models a sense of space
Rather than relying only on flat images, 3D LLMs take data such as point clouds as input. This gives a model information about a scene’s spatial relationships and may help it reason about physics and affordances: what objects are and how they can be used.
That distinction matters for tasks that involve more than naming visible objects. An assistant that needs to navigate or plan actions in a room must relate objects to one another and to locations in the surrounding space. The researchers see potential applications in robotics and embodied AI, where systems need to interact with three-dimensional environments.
Building a bridge between 3D data and language
Training such models requires examples that pair descriptions with 3D scenes. The source article notes that these data sets are limited compared with image-text pairs available on the Web. To address the gap, the team developed prompting techniques for ChatGPT to generate varied descriptions and dialogues about 3D scenes.
The resulting dataset contains over 300,000 3D text examples. It spans several kinds of tasks:
- Labeling objects and parts of a 3D scene
- Answering visual questions
- Breaking a task into steps
- Planning navigation
One example involved asking ChatGPT to describe a 3D bedroom scene and answer questions about objects visible from different angles. That kind of example ties language to a scene that can be viewed from more than one perspective.
Connecting descriptions to coordinates
The team also developed 3D feature extractors. These convert three-dimensional data into a format that works with pre-trained 2D vision-language models, including BLIP-2 and Flamingo. The approach builds on existing models while giving them a way to process 3D input.
A separate 3D localization mechanism links textual descriptions to coordinates in space. This association helps a model represent where a described object or feature is located, rather than treating the description as disconnected text. The researchers also used this mechanism to support efficient training of 3D LLMs with models such as BLIP-2.
What early experiments suggest
Experiments showed that the models could describe 3D scenes in natural language, hold dialogues informed by 3D context, break complex tasks into 3D actions, and connect language with spatial locations. These results point to a way of adding spatial reasoning to AI systems that can already work with language and images.
The findings show capabilities in the tested tasks; they do not establish that an assistant can yet act reliably in any physical setting. The broader idea is to give AI a more useful representation of the spaces it may need to discuss or navigate.
The researchers plan to extend the models to additional tasks and data types, including sound. They also want to apply the work to embodied AI assistants that can interact intelligently with 3D environments. Those aims depend on continuing to connect sensory information, language, and the locations and relationships within a scene.