Monday, August 18, 2025 8/18/2025

Learning Universal Patient Representations with Hierarchical Transformers

Award Number: R01LM012973
ORGANIZATION: NATIONAL LIBRARY OF MEDICINE
OPDIV: NIH
AWARD CLASS: DISCRETIONARY
AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)
PERIOD OF PERFORMANCE START DATE: 02/07/2019
PERIOD OF PERFORMANCE END DATE: 07/31/2027

Group Awards By:

View Award Description

Learning Universal Patient Representations with Hierarchical Transformers - Project Summary Electronic health records contain a wealth of information about patient health status that can be mined for multiple purposes, including clinical research and improved decision-making at the point of care. This information can be represented as structured variables, unstructured text, and images, among other data types. In this work, we develop new models for representing the unstructured text that take advantage of powerful neural models called pre-trained transformers. We propose to make these models usable for much longer texts by adding hierarchical layers to operate over summaries of smaller chunks of text, and shrinking the size of the encoder that operates on smaller chunks. First, we develop a smaller encoder for sentence and paragraph-sized texts, by using a technique called extreme distillation that trains smaller models from the output of larger models. We also propose to pre-train hierarchical models for text, by taking advantage of smaller encoders like that from the first aim. We take advantage of both public and private datasets and experiment with different pre-training tasks and architectures. Our final aim proposes to combine representations learned from text with those from the more mature areas of structured data and images. We design experiments that answer the question of how best to merge these different information sources, and apply them to important clinical classification use cases that are likely to require multiple information sources for accurate performance. Specifically, we address the clinical tasks of predicting injury severity in emergency departments, and predicting diagnosis and prognosis of patients in intensive care units.


Issue Date FY	Funding FY	Legal Entity Name	Legal Entity Address	Legal Entity City	Legal Entity State	Legal Entity Zip Code	Legal Entity COUNTY	Legal Entity COUNTRY	Assistance Listing	Award Code	Budget Year	Action Date	Action Type	Action Amount

Issue Date FY: 2025 ( Subtotal = $571,552 )
2025	2025	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	000	6	8/7/2025	NON-COMPETING CONTINUATION	$571,552
														Subtotal = $571,552

Issue Date FY: 2024 ( Subtotal = $571,552 )
2024	2024	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	003	5	8/5/2024	NON-COMPETING CONTINUATION	$0
2024	2024	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	002	5	7/25/2024	NON-COMPETING CONTINUATION	$571,552
2024	2023	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	001	4	1/31/2024	COMPETING CONTINUATION	$226,072
2024	2021	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	000	3	1/31/2024	NON-COMPETING CONTINUATION	-$226,072
														Subtotal = $571,552

Issue Date FY: 2023 ( Subtotal = $648,153 )
2023	2023	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	002	4	9/15/2023	COMPETING CONTINUATION	$648,153
2023	2021	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	001	3	9/14/2023	NON-COMPETING CONTINUATION	$0
2023	2021	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	000	3	2/8/2023	NON-COMPETING CONTINUATION	$0
														Subtotal = $648,153

Issue Date FY: 2021 ( Subtotal = $366,696 )
2021	2021	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	002	3	5/14/2021	NON-COMPETING CONTINUATION	$18,336
2021	2021	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	001	3	2/24/2021	NON-COMPETING CONTINUATION	$18,335
2021	2021	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	000	3	12/31/2020	NON-COMPETING CONTINUATION	$330,025
														Subtotal = $366,696

Issue Date FY: 2020 ( Subtotal = $367,184 )
2020	2020	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	001	2	5/14/2020	NON-COMPETING CONTINUATION	$36,720
2020	2020	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	000	2	1/15/2020	NON-COMPETING CONTINUATION	$330,464
														Subtotal = $367,184

Issue Date FY: 2019 ( Subtotal = $386,910 )
2019	2019	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	001	1	2/20/2019	NEW	$0
2019	2019	CHILDREN'S HOSPITAL CORPORATION, THE	300 LONGWOOD AVE	BOSTON	MA	02115	SUFFOLK	USA	Medical Library Assistance	000	1	2/7/2019	NEW	$386,910
														Subtotal = $386,910

Grand Total All Awards = $2,912,047

Top

All Categories

About

Search

Reports

Data Submission

Award Information

Learning Universal Patient Representations with Hierarchical Transformers

Award Number: R01LM012973

ORGANIZATION: NATIONAL LIBRARY OF MEDICINE

OPDIV: NIH

AWARD CLASS: DISCRETIONARY

AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)

PERIOD OF PERFORMANCE START DATE: 02/07/2019

PERIOD OF PERFORMANCE END DATE: 07/31/2027

Federal Websites

Department of Health & Human Services

HHS Operating Divisions

HHS Staff Divisions

Download A Document Viewer