Tuesday, January 14, 2025
1/14/2025
Project Summary This project fills a major methods gap that prevents investigators from designing studies with accurate esti- mates of required sample size for multilevel behavioral health implementation studies. Implementation science is essential to achieving NIMH’s mission. An essential step in designing implementation studies is to conduct a statistical power analysis to determine the minimum sample size required to statistically detect effects of interest. Power analyses for implementation research are more complicated because they need to account for (a) patients nested within providers who are nested within organizations or other systems, and (b) scientific aims that typically focus on testing (or at a minimum accounting for) cross-level effects of higher-level (e.g., organization, clinician) implementation determinants or strategies on lower-level (e.g., patient) outcomes. While multilevel power anal- ysis tools are available to accommodate these types of nested studies, the tools require investigators to have prior estimates of three key design parameters to determine the proper sample size for their study —intraclass correlation coefficient (ICC), effect size, and proportion of variance explained by covariates—which are not rou- tinely available from the published literature and cannot be reliably estimated from small pilot studies. Power analyses that use inaccurate estimates of these design parameters are highly likely to be either underpowered, and consequently at-risk of not detecting important effects, or over-powered, and consequently wasteful of lim- ited resources. Lack of reference values for these parameters is a foundational barrier to the field because even small changes in design parameters can dramatically alter the effective sample size from N=300 to N=50. NIMH has funded a large number of implementation studies during the last 10 years (N=140) which provides an opportunity for us to re-access the datasets from these projects to generate accurate estimates of multilevel design parameters for behavioral health implementation studies. We will use NIH-RePorter to identify all NIMH- funded behavioral health implementation studies conducted during the last 10 years and collaborate with PIs to extract design parameters for targeted implementation and clinical outcomes, which we will summarize and pub- lish for the field. We will also generate a predictive model that enables PIs to estimate design parameters tailored to the characteristics of their new studies. Building on our preliminary work within the Penn NIMH ALACRITY Center, this project will (1) generate pooled estimates and ranges of design parameters (i.e., ICCs, effect sizes, covariate R2) needed to accurately estimate sample size in multilevel behavioral health implementation studies, and (2) identify the study characteristics that predict the magnitude of these design parameters. Completion of this work will remove a ubiquitous methodological barrier that undermines the advancement of implementation science in behavioral health. The study will contribute to higher quality, more replicable science, more efficient use of NIMH resources, and higher impact implementation research to improve healthcare quality and well-being for millions of individuals who experience psychiatric disorders each year.
HHS’ Tracking Accountability in Government Grants System (TAGGS) website provides access to detailed descriptions of grants, loans, aggregated direct payments and other types of financial assistance awarded by the HHS Operating Divisions (OPDIVs) and the Office of the Secretary Staff Divisions (STAFFDIVs).