Wednesday, December 17, 2025 12/17/2025

The neural computations supporting hierarchical reinforcement learning

Award Number: R01MH119383
ORGANIZATION: NATIONAL INSTITUTE OF MENTAL HEALTH
OPDIV: NIH
AWARD CLASS: DISCRETIONARY
AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)
PERIOD OF PERFORMANCE START DATE: 04/01/2019
PERIOD OF PERFORMANCE END DATE: 07/31/2025

Group Awards By:

View Award Description

The neural computations supporting hierarchical reinforcement learning - The neural computations supporting hierarchical reinforcement learning - Project Summary. This project explores how humans learn at multiple hierarchical levels in parallel, and how this supports human intelligence. Human decisions are typically hierarchically structured: we make high-level decisions (making a cup of coffee), which constrain lower level decisions (grinding coffee beans, boiling water, etc.), which themselves constrain simpler and simpler decisions and motor actions. This hierarchy in decisions is paralleled by a hierarchy in our representation of our environment: some sensory signals trigger simple decisions (a red light signals a stop), while other signal a broader, more abstract behavioral change (rain signals a set of adaptations when driving). Thus, complex hierarchical structure underlies the way we respond to our environment in seemingly simple, everyday tasks. This ability is supported by the prefrontal cortex, which represents states and decisions at multiple degrees of hierarchical abstraction. My previous work shows that hierarchical representations support transfer and generalization while learning, an ability that artificial agents still struggle to match human performance in. However, how we learn to form these hierarchical representations is poorly understood, despite how crucial it is for human intelligence. The proposed work will examine how multiple, parallel hierarchical loops between prefrontal cortex and the basal ganglia support reinforcement learning at multiple hierarchical levels in parallel, and how this promotes flexible behavior. To this end, we will address three aims: 1. We will show that the same reinforcement learning computations happen in parallel at multiple levels of abstraction, as hypothesized by our computational model of prefrontal- subcortical networks. 2. We will demonstrate that humans partition learning problems into multiple sequential subgoals so they can learn multiple simple strategies instead of one complex strategy, and that reusing these simple strategies promotes fast exploration and learning. 3. We will show that hierarchical learning does not rely exclusively on rewards, but that novelty signals are crucial for identifying subgoals and learning through curiosity. Across all three aims, we will use behavioral experiments in conjunction with computational modeling to characterize how humans learn hierarchically. In addition, we will use EEG and fMRI to identify the neural computations underlying the cognitive systems inferred from behavior and modeling. This project will provide new insights into the computational mechanisms that give rise to learning, and thus provide a better handle on the sources of learning dysfunction observed in many psychiatric diseases, including schizophrenia, depression, anxiety, ADHD, and OCD. Additionally, it will provide new tools, in the form of experimental protocols and precise computational models, for studying learning across populations and species.


Issue Date FY	Funding FY	Legal Entity Name	Legal Entity Address	Legal Entity City	Legal Entity State	Legal Entity Zip Code	Legal Entity COUNTY	Legal Entity COUNTRY	Assistance Listing	Award Code	Budget Year	Action Date	Action Type	Action Amount

Issue Date FY: 2026 ( Subtotal = -$104 )
2026	2023	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 201	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	5	12/11/2025	NON-COMPETING CONTINUATION	-$104
														Subtotal = -$104

Issue Date FY: 2025 ( Subtotal = $0 )
2025	2023	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 201	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	5	4/18/2025	NON-COMPETING CONTINUATION	$0
														Subtotal = $0

Issue Date FY: 2023 ( Subtotal = $379,432 )
2023	2023	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 201	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	5	2/10/2023	NON-COMPETING CONTINUATION	$379,432
														Subtotal = $379,432

Issue Date FY: 2022 ( Subtotal = $380,055 )
2022	2022	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 201	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	4	2/16/2022	NON-COMPETING CONTINUATION	$342,049
2022	2022	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 201	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	001	4	5/23/2022	NON-COMPETING CONTINUATION	$38,006
														Subtotal = $380,055

Issue Date FY: 2021 ( Subtotal = $380,647 )
2021	2021	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 220	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	3	2/23/2021	NON-COMPETING CONTINUATION	$380,647
														Subtotal = $380,647

Issue Date FY: 2020 ( Subtotal = $381,213 )
2020	2020	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 220	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	2	2/21/2020	NON-COMPETING CONTINUATION	$381,213
														Subtotal = $381,213

Issue Date FY: 2019 ( Subtotal = $381,751 )
2019	2019	REGENTS OF THE UNIVERSITY OF CALIFORNIA, THE	1608 4TH ST STE 220	BERKELEY	CA	94710	ALAMEDA	USA	Mental Health Research Grants	000	1	3/15/2019	NEW	$381,751
														Subtotal = $381,751

Grand Total All Awards = $1,902,994

Top

All Categories

About

Search

Reports

Data Submission

Award Information

The neural computations supporting hierarchical reinforcement learning

Award Number: R01MH119383

ORGANIZATION: NATIONAL INSTITUTE OF MENTAL HEALTH

OPDIV: NIH

AWARD CLASS: DISCRETIONARY

AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)

PERIOD OF PERFORMANCE START DATE: 04/01/2019

PERIOD OF PERFORMANCE END DATE: 07/31/2025

Federal Websites

Department of Health & Human Services

HHS Operating Divisions

HHS Staff Divisions

Download A Document Viewer