TL;DR
Satori is an AR headset assistant that models what you know, what you want, and what you’ll do next, so it can offer the right help at the right moment. Without any task-specific scripting, it performed on par with assistants that AR designers built by hand.
Satori in action while making pour-over coffee (left) and the system that powers it (right). Figures from the paper.
Satori in action while making pour-over coffee (left) and the system that powers it (right). Figures from the paper.

The question

Most AR assistants are reactive: they wait for you to ask. A good human helper is proactive. They notice you’re looking for the coffee beans, or that you’re about to skip a step, and they speak up. To do that, an assistant needs a rough model of your mind: what you already know, what you’re trying to achieve, and what you’re about to do.

Today, proactive AR help is usually scripted by hand for each task, which doesn’t scale. Can an AI system infer these mental states on its own and decide when and how to help?

What we did

We first ran two formative studies with 12 AR and design experts to learn what proactive help should look like. We then built Satori on the Belief–Desire–Intention (BDI) model from psychology:

  • Belief: what the user knows about the scene and the task, such as where an object is or how to use it.
  • Desire: the user’s current goal, for example “make coffee.”
  • Intention: the immediate next step, for example “wet the filter.”

Satori runs on a HoloLens. It combines the camera view, the conversation, and the interaction history with a multimodal LLM (GPT-4V) to infer belief, desire, and intention. It then predicts when to step in and what to show: text, a generated illustration, or both.

We tested it with 16 participants on four everyday tasks (arranging flowers, making coffee, cleaning a room, and connecting a Nintendo Switch). Each participant used both Satori and a Wizard-of-Oz baseline, in which AR designers had hand-crafted the guidance for each task.

System overview. Camera, dialogue, and logs feed an LLM that maintains the user's belief, desire, and intention, then decides the timing and content of guidance.
System overview. Camera, dialogue, and logs feed an LLM that maintains the user’s belief, desire, and intention, then decides the timing and content of guidance.

What we found

1. It reads the task and the next step well

Satori identified the user’s task goal with 100% balanced accuracy on two datasets. Its scene understanding (belief) was close to a specialist model (66.5% vs. 69.5% on EgoTaskQA). It predicted the user’s next action with 78% precision. And it chose the same kind of help (text, image, or both) as expert designers 75% of the time.

The interface shows the current goal and the upcoming step (a), and asks for confirmation before moving on (b).
The interface shows the current goal and the upcoming step (a), and asks for confirmation before moving on (b).

2. Users rated it as good as designer-made help

On comprehensibility, helpfulness, timing, and learnability, Satori was statistically not worse than the hand-designed baseline (non-inferiority tests, p < .01). It scored better at reflecting users’ actual intentions and at placing guidance in the right spot (p < .05).

Seven-point ratings for Satori vs. the Wizard-of-Oz baseline across the four tasks.
Seven-point ratings for Satori vs. the Wizard-of-Oz baseline across the four tasks.

3. No extra mental effort

NASA-TLX workload scores showed no difference between Satori and the baseline on any of the six dimensions. Proactive help did not make the task feel harder.

NASA-TLX workload ratings: Satori and the Wizard-of-Oz baseline are indistinguishable.
NASA-TLX workload ratings: Satori and the Wizard-of-Oz baseline are indistinguishable.

My take

BibTeX
@inproceedings{li2025satori,
  title     = {Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modeling},
  author    = {Li, Chenyi and Wu, Guande and Chan, Gromit Yeuk-Yin and Turakhia, Dishita G. and
               Castelo Quispe, Sonia and Li, Dong and Welch, Leslie and Silva, Cl{\'a}udio T. and
               Qian, Jing},
  booktitle = {Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '25},
  year      = {2025},
  publisher = {ACM},
  doi       = {10.1145/3706598.3714188}
}