Preemptive Solving of Future Problems:
Multitask Preplay in Humans and Machines

Wilka Carvalho1, Sam Hall-McMaster2, Honglak Lee3,4, Samuel J. Gershman1,2
1Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, 2Department of Psychology and Center for Brain Science, Harvard University, 3Department of Computer Science & Engineering, University of Michigan, 4LG AI Research
Paper Slides Multitask Preplay Code NiceWebRL
We hypothesize that humans leverage experience on some tasks to preemptively learn solutions for other tasks that were accessible but unpursued. By doing so, they obtain reactive behavior that is adaptive to novel, unpursued tasks---something typically associated with deliberate planning. We operationalize this as Multitask Preplay and present behavioral evidence with a gridworld domain and a 2D minecraft domain. We conclude with AI simulations in the 2D minecraft domain showing that Multitask Preplay improves generalization of complex tasks to new environments that share task co-occurrence structure.

Multitask Preplay

Algorithm overview

Consider someone that has moved to a new neighborhood and visited two coffee shops. Along the way, they observed a grocery store along their route. When they later want to go to the grocery store from their home, what behavior do you think they will exhibit?

Interpolate start reference image.

While people certainly exhibit options 1 and 2 sometimes, we argue that people exhibit option 3 more often than we realize---i.e. that people somehow have access to fast, reactive behavior that can accomplish a novel goal they've previously been exposed to. We hypothesize that this is supported by Multitask Preplay.

Results

Interpolate start reference image.
Interpolate start reference image.
Interpolate start reference image.

Video of person completing generalization task. Left is global view. Right is what they see.

Interpolate start reference image. Interpolate start reference image.
Interpolate start reference image.

Related Links

NiceWebRL is a Python library for quickly making human subject experiments that leverage machine RL environments. It supports making online web experiments with 50+ machine RL environments and was used for all experiments in this paper.

Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior. This is a position paper on why and how we can build generalizable models of human behavior that can scale up to increasingly naturalistic experimental paradigms. This project is an example of doing that: we first predict human behavior in a fully-observable grid-world, and then in a partially-observable 2D minecraft environment.

Predictive representations: building blocks of intelligence. This is a review of predictive representations in machine learning, cognitive science, and neuroscience. We focus on the family of predictive representations defined by the successor representation, but the arguments equally apply to the family defined by generalized value functions (which Multitask Preplay is an example of).

BibTeX

@article{carvalho2025preemptive,
  author    = {Carvalho, Wilka and Hall-McMaster, Sam and Lee, Honglak and Gershman, Samuel J.},
  title     = {Preemptive Solving of Future Problems: Multitask Preplay in Humans and Machines},
  journal   = {arXiv preprint arXiv:2507.05561},
  year      = {2025},
}