An assistant that can help with a task in the physical world needs more than words on a screen. It needs to interpret what someone is doing and understand when guidance would help. Microsoft’s HoloAssist dataset is designed to support research toward that kind of system by pairing first-person recordings of physical tasks with instructions from a human tutor.
Recording tasks from the participant’s point of view
HoloAssist includes more than 160 hours of video recordings and seven sensor streams. The data captures signals such as eye movement, hand and head position, and voice, giving researchers several kinds of information about a person’s actions and attention.
The recordings came from 222 participants who completed 20 different tasks while wearing the Hololens 2 mixed reality headset. A tutor provided instructions during the tasks. Activities ranged from assembling furniture to operating coffee makers and laser scanners.
This setup connects what a participant sees and does with the spoken guidance they receive. That combination may help researchers investigate how an AI assistant could track a task as it unfolds, rather than respond only to an isolated request.
Why physical assistance is a harder problem
According to Microsoft researchers Xin Wang and Neel Joshi, HoloAssist is the first dataset of its kind to combine egocentric videos of people performing physical tasks with associated instructions from a human tutor. The recordings are intended to give researchers examples of how people act and communicate while working through real-world activities.
The researchers say earlier AI assistants remain limited largely to the digital world. They lack the experience and training data needed to help adequately with physical tasks. Without information about the state of a real-world activity, an assistant may not know what a person has already done or when an intervention is needed.
That gap affects more than recognizing objects or following a sequence of steps. A useful assistant would need to connect what it perceives with reasoning about the task and with decisions about whether and when to speak. HoloAssist brings together video, sensor data, and tutor instructions that researchers can use to study those challenges.
From recorded guidance to proactive help
Microsoft’s researchers describe the tutor-led setting as useful for developing assistants that can act proactively and give precise instructions at the right time. In practice, that goal depends on an assistant being able to make sense of an evolving activity and relate its guidance to what the person is doing.
The dataset does not establish that an AI system can already provide this kind of help. Instead, it offers material for experimentation. Researchers can use examples of physical tasks and accompanying instructions to explore how systems might perceive actions, reason about what is happening, and model a task well enough to assist.
If that research progresses, AI assistance could extend into daily activities that involve objects, tools, and physical steps. The source describes this as a potential future direction, rather than a capability HoloAssist has already delivered.
Shared research material and earlier datasets
Microsoft has made HoloAssist available to the scientific community. The company uploaded the code and more than a terabyte of training materials to GitHub, allowing researchers to conduct further experiments with the dataset.
HoloAssist follows earlier efforts to collect first-person video for AI research. Meta provided more than 2,200 hours of first-person perspective footage in 2021 through a dataset called Ego4D, which will also be used to develop an AI assistance system for AR devices.
In 2022, seven hours of first-person video followed in the Project Aria Pilot Dataset. Those videos were recorded using an early prototype of the AR glasses Meta hopes to build. The prototype included cameras for filming but had no displays.
Together, these efforts illustrate the kind of material researchers are assembling to study AI assistance beyond digital tasks. HoloAssist adds recordings that pair physical activity with a tutor’s instructions, creating a resource for examining how an assistant might follow a task and provide guidance in context.