Multitask Preplay · Under review at PNAS · 2025

Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines

During offline replay, humans and AI agents simulate pursuit of goals they did not pursue — caching solutions into a shared predictive representation that explains a counterintuitive human bias and improves RL generalization to 10,000 unseen worlds.

Wilka Carvalho1Sam Hall-McMaster2Honglak Lee3,4Samuel J. Gershman1,2
1Kempner Institute · Harvard 2Psychology & CBS · Harvard 3CSE · Michigan 4LG AI Research
Paper (PDF) arXiv Code Data Thread Slides
Demo. One mechanism. Two puzzles. While pursuing one task in our 2D Minecraft (Craftax) environment, an agent encounters accessible goals it doesn't pursue. During offline replay, it preplays — counterfactually simulates — pursuing those goals, caching predictions for them. When a new task arrives, behavior on it is fast and reactive, as if the route had already been rehearsed.
TLDR

People are faster at new tasks when they reuse an old path — even when the reused path is longer than the optimal shortcut. We argue this happens because, during offline replay, humans and AI agents simulate goals they didn't pursue — caching solutions into a shared predictive representation. The same algorithm explains a behavioral bias in people and scales an RL agent to 10,000 unseen Craftax worlds.

  1. Algorithm. Multitask Preplay: replay one task; preplay the goals you observed but didn't pursue; cache the resulting solutions into a shared predictive representation.
  2. Behavioral evidence. Across five behavioral experiments in grid-worlds and 2D Minecraft (Craftax) — n=100 each, with preregistered predictions — people behave as if they have preemptively rehearsed goals before they encounter them: lower response times at familiar junctures pointing toward an unannounced goal, and partial reuse of old paths even when a shorter route is available.
  3. AI scaling. Beats Dyna, Universal Value Function Approximators (UVFA), Universal Successor Feature Approximators (USFA), and Hindsight Experience Replay (HER) on transfer to 10,000 held-out Craftax environments.
  4. Bridging. One computational idea — offline counterfactual simulation — connects a human behavioral bias to RL generalization, a template for AI and cognitive science to keep informing each other.
The Idea

Preplay the goals you didn't pursue.

Do RL agents really need to replan from scratch for every new goal? Why do rodents "preplay" goals they haven't pursued?

While searching for coffee in a new neighborhood, you may come across gyms, grocery stores, and parks — enabling you to quickly find these locations later when you need them. You weren't told to look for the grocery store. But when the goal becomes "go to the grocery store," your behavior is fast and reactive, as if you'd planned the route already.

We hypothesize this happens because, during offline replay, you imagine pursuing goals you encountered along the way but didn't actually pursue. Those counterfactual trajectories cache solutions into a single value function that covers many goals. We call it Multitask Preplay — a nod to hippocampal preplay, where place cells fire for places the animal hasn't been yet (Dragoi & Tonegawa, 2011). The underlying RL idea goes back to Kaelbling (1993): use one experience to learn about many goals. We wanted to see what it looks like at scale.

Multitask Preplay generalizes Dyna (Sutton, 1990), which only caches a value for the goal you pursued. It keeps Dyna's model-based simulation and adds what successor features (Barreto et al., 2017) do well — predictive representations that span goals.

Schematic of Multitask Preplay. While pursuing goal g, the agent observes accessible goal g' along the trajectory; offline, it preplays a counterfactual rollout to g' and caches predictions for both into a shared value function.
Figure 1. Multitask Preplay overview. (1) The agent pursues a task and observes other accessible goals along the way. (2) During offline replay, the agent counterfactually simulates pursuit of those other goals. (3) Predictions for all goals are cached into a shared representation. (4) When an unannounced goal becomes the actual task, behavior is fast and reactive — no new experience or planning required.
Two-axis categorization of RL methods. Top: a 2x3 grid mapping number-of-predictions-per-state (single reward vs. multiple) against compute strategy (model-free, model-based background learning, decision-time planning). Multitask Preplay (Ours) sits in the multi-prediction model-based-background cell. Bottom: gridworld behavior for Q-learning, Successor Features / HER, Dyna, and Multitask Preplay, showing what each caches when the agent has experienced the path to a coffee shop while observing food and grocery store along the way.
Figure 2. Where Multitask Preplay fits among RL methods. Top: Two axes — number of predictions per state (1 reward vs. multiple) × compute strategy (model-free, model-based background learning, decision-time planning). Yellow methods cache the results of computation; purple methods rely on model-based planning. Bottom: Behavior in a gridworld where the agent walked to a coffee shop while passing food and a grocery store. (a) Q-learning caches a single reward prediction along the experienced path. (b) Successor Features / HER cache predictions for multiple goals — but only those whose features were experienced. (c) Dyna simulates extra coffee-shop trajectories but still refines only the single coffee-shop prediction. (d) Multitask Preplay simulates trajectories to accessible-but-unpursued goals (food, groceries), caching a predictive map that covers all three.
Behavioral Evidence

People behave as if they've rehearsed goals before they encounter them.

We trained people on one set of tasks, then tested them on different ones. In principle, you'd rehearse tasks you expect to face. In practice — across five behavioral experiments in grid-worlds (JaxMaze) and 2D Minecraft (Craftax), n=100 each, with preregistered predictions — people behave as if they've rehearsed goals they were never told about.

The headline finding: the same person is faster at the same novel task when starting from a familiar location than from an unfamiliar one — even when both starts are equally far from the goal. Under Multitask Preplay this is expected: someone who passed that juncture during training would have preplayed routes from it to nearby goals, so the route is already cached when the test arrives. The alternatives predict no such advantage — under decision-time planning, for example, response times should be approximately equal from either start. Only Multitask Preplay predicts the familiar-juncture speedup.

Demo. A subject performing a trial in our juncture-spawn experiment. Spawned at a familiar juncture from training, the subject is asked to reach a novel goal. Their first response from the familiar juncture is faster than when the same subject is spawned at an unfamiliar location the same distance from the goal.
Four panels (I, J, K, L). I-K: JaxMaze layouts showing the same eval target reached from a familiar juncture (Near, Known), the same juncture with the eval goal hidden (Near, Unknown), and a novel distance-matched location (Far, Known). L: bar chart of delta log first-RT for each spawn condition; novel-spawn is significantly slower than juncture-spawn for both known and unknown evaluation goals.
Figure 3. Juncture-spawn experiment (JaxMaze). Subjects are spawned at one of three locations during evaluation: (I) a familiar juncture from training, with the test goal announced; (J) the same familiar juncture, but the test goal is unannounced and must be inferred from the available cues; (K) an unfamiliar location, distance-matched to the goal. (L) First response times (Δ log RT) are significantly higher from unfamiliar locations than from familiar junctures (***) — subjects are faster from familiar junctures even when the test goal is unannounced. Subjects behave as if they have rehearsed the path to those goals before they encounter them as the announced task.

The juncture-spawn advantage is one signature; partial path reuse is another. Give a person a new goal near an old route, and they don't take the optimal shortcut — they partially reuse the trained path. We observe this even when the reused path is longer than the available shortcut: in our path-reuse experiment, the reused path runs 61 steps against a 55-step shortcut, and subjects still partially reuse the trained path 62.8% of the time, with lower step-by-step response times on the reused segment.

To test whether this is generic habit or something specific to multi-goal preplay, we benchmarked humans against a panel of RL methods in Craftax. Humans and Multitask Preplay cluster at high success and high path reuse; Dyna, UVFA, USFA, and HER generalize but don't reuse paths the way humans do.

Three panels (A, B, C). A: a procedurally generated 2D Minecraft (Craftax) overworld with the agent spawn, training stones marked by stars, and an evaluation stone visible nearby. B: the agent's partial view (foreground panel) and inventory bar. C: scatter plot of path-reuse rate (X) vs. success rate (Y); humans (blue: known eval goal; orange: unknown eval goal) cluster at high success and high path-reuse, with Multitask Preplay alongside; Dyna, Universal Value Function, Universal Landmark Successor Features, and Hindsight Experience Replay all sit far below — they generalize but don't reuse paths.
Figure 4. 2D Minecraft (Craftax) behavioral experiment. (A) Subjects learn to obtain training stones across procedurally generated maps; an evaluation stone is visibly nearby in some maps. (B) During evaluation, subjects only see a partial view of the world plus their inventory. (C) Generalization success vs. partial path reuse, by model. Humans cluster at high success and high partial path-reuse — both when the eval goal is known (blue) and unknown (orange). Of all RL methods tested, only Multitask Preplay lands in the same region. Dyna, UVFA, USFA, and HER all fall well below: they generalize but don't partially reuse paths the way humans do.
Results

Only humans and Multitask Preplay survive the train-to-test cliff.

The result we found most telling: in Craftax, every baseline — Dyna, UVFA, USFA, HER — reaches near-perfect success during training. At test, only humans and Multitask Preplay transfer. Every other method collapses.

Train and test panels side-by-side. All methods reach near-perfect performance during training. At test, only Human and Multitask Preplay bars stand; all other RL baselines fall to near zero.
Figure 5. Train vs. test on Craftax. Train (left): all methods, including humans, reach near-ceiling performance. Test (right): only humans and Multitask Preplay transfer. Every other RL baseline collapses. Humans and AI are evaluated on the same test set.

One agent, 10,000 unseen worlds.

This same algorithm scales further. Trained on a fixed set of environments, Multitask Preplay generalizes to 10,000 unseen Craftax worlds — and its test performance keeps climbing as training environments are added, while Dyna and model-free methods plateau at lower performance.

Generalization performance to 10,000 unique held-out environments as a function of training environment count. Multitask Preplay continues to scale; Dyna and model-free baselines plateau.
Figure 6. AI generalization to 10,000 held-out Craftax environments. Multitask Preplay continues to improve as the training-environment count increases. Dyna (1M training steps) and model-free methods (10M training steps) plateau long before. Each point: mean ± SE across 5 model initializations.
Two signatures in different systems, one computational idea underneath.

The same trick that explains a behavioral bias in people also scales an RL agent to unseen worlds. We're hopeful this is a template for how AI and cognitive science can keep informing each other: a computational-level claim that constrains both the algorithm space (in RL) and the hypothesis space (in cognitive science).

Limitations

What this work doesn't do.

You need a reasonably accurate world model to preplay with. Multitask Preplay assumes access to a model that can roll out trajectories toward counterfactual goals during offline replay. We use ground-truth simulators in this paper. Whether the result holds when the world model itself is learned online — and how preplay interacts with a noisy or biased model — is not established here.

Tested on 2D environments with discrete tasks. Our results span gridworlds and 2D Minecraft. Whether the same algorithm scales to high-dimensional continuous control or to compositionally complex task structures — say, many interrelated tasks in unseen 3D homes — is the natural next step, but it's not what we showed.

Behavioral evidence is consistent with preplay, not unique to it. Our behavioral predictions are derived from Multitask Preplay, and the data are consistent with those predictions. Other algorithms that share its core ingredient — off-policy multi-goal value caching with TD learning — may make similar predictions. What we can say: among the baselines we compared, Multitask Preplay alone matches both human path-reuse rates and the train/test cliff.

What's Next

Learned world models, then unseen homes.

Next we want to see what happens when the world model is learned online — for example, with MuZero — and to scale to domains like Habitat, where agents need to perform interrelated tasks in unseen homes.

Scaling to Habitat will lean on two corrections we developed along the way: off-task Q(λ) and conservative all-goals learning. They're what let us match the human data — and what we think will unlock Habitat-class transfer.

Methods (for the technically curious)

Off-task Q(λ). Propagates multi-step returns across goals the agent didn't pursue — a correction on Peng's Q(λ), applied on the goal axis instead of the policy axis. The trace is cut when the greedy actions for the off-task goal diverge from the actions actually taken, preventing erroneous backup of value from on-task rewards that don't apply to the off-task goal.

Conservative all-goals learning. Keeps value estimates from drifting when extrapolating to unseen goals. Penalizes Q-values for actions absent from the data, suppressing inflated estimates while still propagating value from the true off-task reward.

A 3×3 grid comparing three losses (L_Qλ, L_OQλ, L_COQλ) across three trajectory cases. Top row: when greedy off-task actions match online actions, all three losses correctly back up value (green check). Middle row: when greedy off-task actions diverge from online actions, plain Q(λ) erroneously propagates on-task reward (red X) while off-task Q(λ) and the conservative variant cut the trace correctly. Bottom row: when greedy off-task actions are suboptimal for on-task data (overestimated Q-values), plain Q(λ) backs up the inflated value (red X), off-task Q(λ) backs up only from the inflated estimate (red X), and only the conservative variant suppresses the overestimate while still propagating from the true off-task reward (green check).
Figure M1. Why both corrections are needed. Each row depicts a trajectory τ on the left, with backup paths under three losses on the right: Q(λ), off-task Q(λ), and conservative off-task Q(λ). Top: when the off-task greedy policy matches the online actions, all three losses correctly propagate value from the rewarding state. Middle: when those greedy actions diverge, plain Q(λ) erroneously backs up on-task reward; off-task Q(λ) and its conservative variant cut the trace and propagate only from the off-task rewarding state. Bottom: when an unseen action's Q-value is overestimated, both plain and off-task Q(λ) back up from that inflated value; only the conservative variant suppresses inflated estimates while still propagating from the true reward.

Both corrections are drop-in on a Dyna backbone. Full derivations + ablations in §Methods of the paper. Reference implementation in github.com/wcarvalho/multitask_preplay.

Cite

BibTeX

@article{carvalho2025preemptive,
  author    = {Carvalho, Wilka and Hall-McMaster, Sam and Lee, Honglak and Gershman, Samuel J.},
  title     = {Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines},
  journal   = {arXiv preprint arXiv:2507.05561},
  year      = {2025},
}