For an AI learning to play a game, a manual can provide something that trial and error may take a long time to discover: what objects matter and which interactions are useful. Researchers have used written instructions to help an agent learn an Atari game with far fewer training frames, offering a way to give reinforcement learning a clearer signal.
Why Atari Skiing takes so long to learn
DeepMind scientists unveiled Agent57 in March 2020. It was the first deep reinforcement learning model to outperform humans across all 57 Atari 2600 games. Yet even that achievement required enormous training for some games.
In Skiing, the agent must avoid trees on a slope. Agent57 needed 80 billion training frames for the game. At 30 frames per second, that amount of play would take a human nearly 85 years. The challenge is not simply making moves: the agent must work out which actions lead to progress from the game itself.
This is especially difficult in games with sparse rewards. The player may need to carry out complex behavior before the game provides a reward. An agent learning only through trial and error can therefore go a long time without a useful signal about whether its actions are helping.
Turning instructions into learning signals
In a paper called “Read and Reap the Rewards,” researchers from Carnegie Mellon University, Ariel University, and Microsoft Research describe a framework that gives the agent information from human-written instructions. Those instructions might come from a game manual or another source, including Wikipedia.
The framework has two parts. A QA Extraction module asks questions about the instructions and extracts relevant answers, organizing information that may otherwise be long or repetitive. A Reasoning module then uses that information to assess interactions between the agent and objects in the game.
When it recognizes an event described by the instructions, the framework assigns a help reward. That reward is passed to an A2C-RL agent, or Advantage Actor Critic agent, to provide additional guidance during learning. The instructions do not replace the game environment; they help the agent interpret what is happening within it.
Fewer frames, with a key caveat
The researchers report that using instructions can reduce the number of training frames by a factor of 1,000. In an interview with New Scientist, first author Yue Wu described a speed-up by a factor of 6,000. For Skiing, the article reports a reduction from 80 billion frames to as little as 13 million, or five days.
The framework improved performance in four Atari games with sparse rewards. These results point to the value of supplying an agent with relevant context: if it can connect an instruction to an event on screen, it may receive guidance before the game’s own reward arrives.
But making that connection is a central challenge. Instructions can be lengthy and redundant, and important information may be implicit. A system must both extract useful details and reason about how they relate to what is happening in the game.
From Atari screens to other settings
Object recognition is another limitation identified by the researchers. The agent needs to recognize objects in the game to evaluate its interactions with them. The researchers note that modern games can provide object ground truth, which addresses this issue in that setting.
They also point to advances in multimodal video-language models as a possible route to more reliable object recognition, potentially replacing that part of the current framework. For real-world applications, advanced computer vision algorithms could help with the same problem.
The broader idea is straightforward: written guidance may make reinforcement learning more efficient when an agent struggles to discover useful behavior from rewards alone. The Atari results do not establish how well the method will work elsewhere, but they show how manuals can serve as a source of structured hints—provided the system can connect those hints to what it sees.