TL;DR
A transformer can learn to learn. After exploring a new maze just once, it heads almost straight to the hidden goal on the next try, using only the experience in its context window. Its internal representations and attention show how.
The agent explores a new maze once, then exploits what it learned to go straight to the goal. When the walls change, it has to think before acting. Right: the transformer reads its experience and outputs actions.
The agent explores a new maze once, then exploits what it learned to go straight to the goal. When the walls change, it has to think before acting. Right: the transformer reads its experience and outputs actions.

The question

People adapt fast. Drop us in a new building and after one walk-through we can find the exit. This ability to get better at learning itself is called meta-learning. Large language models show something similar, called in-context learning: they solve new problems from a few examples in the prompt, without retraining.

Recurrent networks are known to meta-learn in reinforcement-learning tasks. Can a transformer do the same in a planning task? And if so, what representations and attention patterns make it possible?

What we did

Task. We used a 4×4 maze navigation task (after Jensen et al., 2024). In each 100-step episode the agent must find a hidden goal. Every time it reaches the goal, it restarts from a new position while the goal and walls stay put. Across episodes, the goal and walls change. Rewards are +1 for reaching the goal and –1 for hitting a wall. An agent that meta-learns should explore once, remember where the goal is, and then go straight there.

The maze task: explore in trial 1, exploit in later trials, and adapt when the goal and walls change across episodes.
The maze task: explore in trial 1, exploit in later trials, and adapt when the goal and walls change across episodes.

Model. We framed reinforcement learning as sequence prediction. A decision transformer reads a sequence of (state, action, reward, next state) tokens and predicts the best next action. We trained it with supervised learning on 10,000 sequences of random-walk experience. At test time, its context holds its own earlier trials in the same maze.

Supervised decision transformer: trained on random-walk experience to predict the optimal action, then run on its own experience in a new maze.
Supervised decision transformer: trained on random-walk experience to predict the optimal action, then run on its own experience in a new maze.

What we found

1. One trial is enough

In a new maze, the agent first explores almost every square. After a single trial, the number of steps it needs to reach the goal drops sharply to near-optimal. That is one-shot learning, happening entirely in context.

Agent behavior in a new maze (left) and average steps-to-goal across trials (right): a sharp drop after the first trial.
Agent behavior in a new maze (left) and average steps-to-goal across trials (right): a sharp drop after the first trial.

2. The goal’s location can be read out of the network

We trained simple decoders to read the goal location from the transformer’s internal activations (its residual stream). Decoding accuracy rose as the episode went on, and different layers carried this information to different degrees. The network builds an internal map of where the goal is.

Accuracy of decoding the goal location from each layer's residual stream, over time steps in the episode.
Accuracy of decoding the goal location from each layer’s residual stream, over time steps in the episode.

3. Attention switches from exploring to exploiting

The attention map splits into an exploration phase and an exploitation phase. In the exploitation phase, attention concentrates on particular past steps, such as those two steps away from the goal. The model selectively attends to relevant past experience to decide what to do next.

Attention from the last token in the final layer: a clear switch between the explore and exploit phases, with focused attention on steps near the goal.
Attention from the last token in the final layer: a clear switch between the explore and exploit phases, with focused attention on steps near the goal.

My take

BibTeX
@unpublished{li2024learning,
  title  = {Learning to Reinforcement Learn with Transformer},
  author = {Li, Chenyi and Kumar, Sreejan and Mattar, Marcelo},
  note   = {Manuscript in preparation},
  year   = {2024}
}