Expanding Translational Science in Arkansas - Clinical prediction models have increasingly been developed due to advances in big data, computational speed, and statistical techniques. Such models are also growing for pregnancy-relevant outcomes such as maternal mortality, severe maternal morbidity, diabetes/hypertension, and delivery types of stillbirths or ectopic pregnancies. However, these models largely lack external validation and also suffer from heterogeneous performance and poor reporting. More importantly, evidence is lacking on these prediction models’ performance across groups such as younger/older pregnant women, privately/publicly insured, and those residing in rural/urban areas. Research on documentation of differential model performance, their plausible reasons, and the methods to correct them is lacking, particularly in pregnancy, where there is high variability in health outcomes. This administrative supplement to 1UM1TR004909 proposes to investigate performance of two types of risk scores across two maternal age groups (younger/older), two insured categories (public/private), and two rural/urban categories (rural/urban residence). The following two aims will be pursued: 1) estimate model performance (true positive/false positive rates) of the in-built Epic risk scores for diabetes-related hospitalization or emergency department visit across maternal age, insurance status, and rural/urban groups of pregnant women; and 2) estimate algorithm performance measures of a published prediction model for obstetric comorbidity risk score across maternal age, insurance status, and rural/urban groups of pregnant women. These studies will be conducted using a national database of electronic medical records from Epic, known as the Epic Cosmos database, from 2019 through 2024. The database is a result of community collaboration between health systems using Epic EMR system and encompasses more than 289 million individuals from 1,626 hospitals and 37,700 clinics. In addition to utilizing previously developed model performance measures that assess true/false positive rates and model calibration, this project will also introduce novel approaches to probe for differential performance of risk scores, such as covariate-adjusted true/false positive rates, adjustment for treatment drop-in, and the application of gap-closing estimand framework. The project will also adapt two in-processing methods (a regularization technique and constrained learning for misclassification) to post-processing setting, contributing to methodological advances in algorithm performance. As these risk scores could drive preventive and treatment decisions, poorly calibrated risk scores could perpetuate variability in health outcomes. Examining these risk scores for poor performance, inspecting the likely causes, and correcting them would optimize aligning clinical decisions with the true risk, thereby improving health outcomes among all pregnant women.
