Project Summary
This project fills a major methods gap that prevents investigators from designing studies with accurate esti-
mates of required sample size for multilevel behavioral health implementation studies. Implementation science
is essential to achieving NIMH’s mission. An essential step in designing implementation studies is to conduct a
statistical power analysis to determine the minimum sample size required to statistically detect effects of interest.
Power analyses for implementation research are more complicated because they need to account for (a) patients
nested within providers who are nested within organizations or other systems, and (b) scientific aims that typically
focus on testing (or at a minimum accounting for) cross-level effects of higher-level (e.g., organization, clinician)
implementation determinants or strategies on lower-level (e.g., patient) outcomes. While multilevel power anal-
ysis tools are available to accommodate these types of nested studies, the tools require investigators to have
prior estimates of three key design parameters to determine the proper sample size for their study —intraclass
correlation coefficient (ICC), effect size, and proportion of variance explained by covariates—which are not rou-
tinely available from the published literature and cannot be reliably estimated from small pilot studies. Power
analyses that use inaccurate estimates of these design parameters are highly likely to be either underpowered,
and consequently at-risk of not detecting important effects, or over-powered, and consequently wasteful of lim-
ited resources. Lack of reference values for these parameters is a foundational barrier to the field because even
small changes in design parameters can dramatically alter the effective sample size from N=300 to N=50.
NIMH has funded a large number of implementation studies during the last 10 years (N=140) which provides
an opportunity for us to re-access the datasets from these projects to generate accurate estimates of multilevel
design parameters for behavioral health implementation studies. We will use NIH-RePorter to identify all NIMH-
funded behavioral health implementation studies conducted during the last 10 years and collaborate with PIs to
extract design parameters for targeted implementation and clinical outcomes, which we will summarize and pub-
lish for the field. We will also generate a predictive model that enables PIs to estimate design parameters tailored
to the characteristics of their new studies. Building on our preliminary work within the Penn NIMH ALACRITY
Center, this project will (1) generate pooled estimates and ranges of design parameters (i.e., ICCs, effect sizes,
covariate R2) needed to accurately estimate sample size in multilevel behavioral health implementation studies,
and (2) identify the study characteristics that predict the magnitude of these design parameters. Completion of
this work will remove a ubiquitous methodological barrier that undermines the advancement of implementation
science in behavioral health. The study will contribute to higher quality, more replicable science, more efficient
use of NIMH resources, and higher impact implementation research to improve healthcare quality and well-being
for millions of individuals who experience psychiatric disorders each year.