
The question
People adapt fast. Drop us in a new building and after one walk-through we can find the exit. This ability to get better at learning itself is called meta-learning. Large language models show something similar, called in-context learning: they solve new problems from a few examples in the prompt, without retraining.
Recurrent networks are known to meta-learn in reinforcement-learning tasks. Can a transformer do the same in a planning task? And if so, what representations and attention patterns make it possible?
What we did
Task. We used a 4×4 maze navigation task (after Jensen et al., 2024). In each 100-step episode the agent must find a hidden goal. Every time it reaches the goal, it restarts from a new position while the goal and walls stay put. Across episodes, the goal and walls change. Rewards are +1 for reaching the goal and –1 for hitting a wall. An agent that meta-learns should explore once, remember where the goal is, and then go straight there.

Model. We framed reinforcement learning as sequence prediction. A decision transformer reads a sequence of (state, action, reward, next state) tokens and predicts the best next action. We trained it with supervised learning on 10,000 sequences of random-walk experience. At test time, its context holds its own earlier trials in the same maze.

What we found
1. One trial is enough
In a new maze, the agent first explores almost every square. After a single trial, the number of steps it needs to reach the goal drops sharply to near-optimal. That is one-shot learning, happening entirely in context.

2. The goal’s location can be read out of the network
We trained simple decoders to read the goal location from the transformer’s internal activations (its residual stream). Decoding accuracy rose as the episode went on, and different layers carried this information to different degrees. The network builds an internal map of where the goal is.

3. Attention switches from exploring to exploiting
The attention map splits into an exploration phase and an exploitation phase. In the exploitation phase, attention concentrates on particular past steps, such as those two steps away from the goal. The model selectively attends to relevant past experience to decide what to do next.

My take
Links
BibTeX
@unpublished{li2024learning,
title = {Learning to Reinforcement Learn with Transformer},
author = {Li, Chenyi and Kumar, Sreejan and Mattar, Marcelo},
note = {Manuscript in preparation},
year = {2024}
}