Thursday, July 17, 2025 7/17/2025

PsyRAT: Extensible Open-Source Software for Applying Generalizability Theory to Assess Psychometric Reliability of Trial-Wise Scores and Optimize Tasks for RDoC

Award Number: R01MH128208
ORGANIZATION: NATIONAL INSTITUTE OF MENTAL HEALTH
OPDIV: NIH
AWARD CLASS: DISCRETIONARY
AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)
PERIOD OF PERFORMANCE START DATE: 08/04/2022
PERIOD OF PERFORMANCE END DATE: 06/30/2027

Group Awards By:

View Award Description

PsyRAT: Extensible Open-Source Software for Applying Generalizability Theory to Assess Psychometric Reliability of Trial-Wise Scores and Optimize Tasks for RDoC - PROJECT SUMMARY/ABSTRACT Neural biomarkers are an important focus of the NIMH Research Domain Criteria (RDoC) initiative, and they are increasingly used in the context of genomic studies and clinical trials. The use of biomarkers with strong psychometric reliability increases the likelihood of finding replicable effects, improves the validity of their interpretation, and decreases the likelihood of missing real phenomena. Although fundamental psychometric principles have long been a prominent concern among studies that use self-report measures, these principles are underappreciated in studies of psychopathology that use biological measures. This lack of attention to reliability limits the more widespread application of biomarkers in psychopathology and likely contributes to replication problems. Generalizability theory is a multifaceted framework for identifying sources of measurement error, and this framework is uniquely suited to assessing the reliability of biological measures and to optimizing tasks for reliability. A critical need exists for tractable software to facilitate the application of generalizability theory to time-frequency electroencephalography (EEG), event-related potentials (ERPs), facial electromyography (EMG), and electrodermal activity (EDA). The objective of this project in response to PAR- 18-930 on measurement tool development for RDoC is to (i) develop an extensive treatment of generalizability theory for psychopathology researchers, (ii) develop accessible software to implement it, (iii) show how to apply these resources to optimize paradigms for individual-differences research, and (iv) disseminate the software with a user-friendly guide. This project will facilitate the routine evaluation of reliability through these specific aims: 1) Design and implement generalizability theory formulas for evaluating group- and subject-level reliability for paradigm optimization; 2) Develop software to implement these formulas with data from widely used psychophysiological software; 3) Apply results to optimize three commonly studied tasks; and 4) Develop online educational material on the application of these resources to novel paradigms and measures. This research project is innovative, because it represents a substantive departure from standard practice by shifting the focus to the reliability of data from individuals, rather than groups, to identify sources of measurement error and minimize their impact. This work promotes best practices in reporting psychometric properties of biological measures and is applicable to data from any task with trial-wise scores. The resulting open-source toolbox, the Psychophysiologist’s Reliability Analysis Toolbox (PsyRAT), can facilitate guidelines for optimizing paradigms, making decisions about individual-subject data, and grounding individual-differences questions (central to clinical research, especially for applications in precision medicine) in measures of reliability. The proposed process for guiding biomarker evaluation through high-quality psychometrics will pave the way for better selection of biomarkers and task development, ultimately improving the clinical utility of these biomarkers.


Issue Date FY	Funding FY	Legal Entity Name	Legal Entity Address	Legal Entity City	Legal Entity State	Legal Entity Zip Code	Legal Entity COUNTY	Legal Entity COUNTRY	Assistance Listing	Award Code	Budget Year	Action Date	Action Type	Action Amount

Issue Date FY: 2025 ( Subtotal = $580,944 )
2025	2025	UNIVERSITY OF SOUTH FLORIDA	4202 E FOWLER AVE	TAMPA	FL	33620	HILLSBOROUGH	USA	Mental Health Research Grants	000	4	6/13/2025	NON-COMPETING CONTINUATION	$580,944
														Subtotal = $580,944

Issue Date FY: 2024 ( Subtotal = $589,164 )
2024	2024	UNIVERSITY OF SOUTH FLORIDA	4202 E FOWLER AVE	TAMPA	FL	33620	HILLSBOROUGH	USA	Mental Health Research Grants	000	3	5/31/2024	NON-COMPETING CONTINUATION	$589,164
														Subtotal = $589,164

Issue Date FY: 2023 ( Subtotal = $576,175 )
2023	2023	UNIVERSITY OF SOUTH FLORIDA	4202 E FOWLER AVE	TAMPA	FL	33620	HILLSBOROUGH	USA	Mental Health Research Grants	001	2	6/21/2023	NON-COMPETING CONTINUATION	$576,175
2023	2022	UNIVERSITY OF SOUTH FLORIDA	4202 E FOWLER AVE	TAMPA	FL	33620	HILLSBOROUGH	USA	Mental Health Research Grants	000	1	11/17/2022	NEW	$0
														Subtotal = $576,175

Issue Date FY: 2022 ( Subtotal = $643,035 )
2022	2022	UNIVERSITY OF SOUTH FLORIDA	4202 E FOWLER AVE	TAMPA	FL	33620	HILLSBOROUGH	USA	Mental Health Research Grants	001	1	9/8/2022	NEW	$0
2022	2022	UNIVERSITY OF SOUTH FLORIDA	4202 E FOWLER AVE	TAMPA	FL	33620	HILLSBOROUGH	USA	Mental Health Research Grants	000	1	8/4/2022	NEW	$643,035
														Subtotal = $643,035

Grand Total All Awards = $2,389,318

Top

All Categories

About

Search

Reports

Data Submission

Award Information

PsyRAT: Extensible Open-Source Software for Applying Generalizability Theory to Assess Psychometric Reliability of Trial-Wise Scores and Optimize Tasks for RDoC

Award Number: R01MH128208

ORGANIZATION: NATIONAL INSTITUTE OF MENTAL HEALTH

OPDIV: NIH

AWARD CLASS: DISCRETIONARY

AWARD ACTIVITY TYPE: SCIENTIFIC/HEALTH RESEARCH (INCLUDES SURVEYS)

PERIOD OF PERFORMANCE START DATE: 08/04/2022

PERIOD OF PERFORMANCE END DATE: 06/30/2027

Federal Websites

Department of Health & Human Services

HHS Operating Divisions

HHS Staff Divisions

Download A Document Viewer