The neural computations supporting hierarchical reinforcement learning - Project Summary.
This project explores how humans learn at multiple hierarchical levels in parallel, and how this supports human
intelligence. Human decisions are typically hierarchically structured: we make high-level decisions (making a
cup of coffee), which constrain lower level decisions (grinding coffee beans, boiling water, etc.), which
themselves constrain simpler and simpler decisions and motor actions. This hierarchy in decisions is paralleled
by a hierarchy in our representation of our environment: some sensory signals trigger simple decisions (a red
light signals a stop), while other signal a broader, more abstract behavioral change (rain signals a set of
adaptations when driving). Thus, complex hierarchical structure underlies the way we respond to our
environment in seemingly simple, everyday tasks. This ability is supported by the prefrontal cortex, which
represents states and decisions at multiple degrees of hierarchical abstraction. My previous work shows that
hierarchical representations support transfer and generalization while learning, an ability that artificial agents
still struggle to match human performance in. However, how we learn to form these hierarchical
representations is poorly understood, despite how crucial it is for human intelligence. The proposed work will
examine how multiple, parallel hierarchical loops between prefrontal cortex and the basal ganglia support
reinforcement learning at multiple hierarchical levels in parallel, and how this promotes flexible behavior. To
this end, we will address three aims: 1. We will show that the same reinforcement learning computations
happen in parallel at multiple levels of abstraction, as hypothesized by our computational model of prefrontal-
subcortical networks. 2. We will demonstrate that humans partition learning problems into multiple sequential
subgoals so they can learn multiple simple strategies instead of one complex strategy, and that reusing these
simple strategies promotes fast exploration and learning. 3. We will show that hierarchical learning does not
rely exclusively on rewards, but that novelty signals are crucial for identifying subgoals and learning through
curiosity. Across all three aims, we will use behavioral experiments in conjunction with computational modeling
to characterize how humans learn hierarchically. In addition, we will use EEG and fMRI to identify the neural
computations underlying the cognitive systems inferred from behavior and modeling. This project will provide
new insights into the computational mechanisms that give rise to learning, and thus provide a better handle on
the sources of learning dysfunction observed in many psychiatric diseases, including schizophrenia,
depression, anxiety, ADHD, and OCD. Additionally, it will provide new tools, in the form of experimental
protocols and precise computational models, for studying learning across populations and species.