Estimating the External Validity of Randomized Controlled Trials: A Comparison of Morbidity and Mortality Between an RCT and an Observational Study in Botswana

Abstract Background: The external validity of the randomized controlled trial (RCT) refers to the extent to which the results of the RCT apply to the non-trial population. External validity of RCTs is impacted by the representativeness of the population enrolled and differences in the standard of care between the trial and non-trial setting. In spite of these concerns that have been historically acknowledged, external validity of RCTs is difficult to study empirically given confounding due to time and population under study. Here, we compared the outcomes of mortality and hospitalization between an RCT and an observational study that concurrently enrolled HIV-exposed uninfected (HEU) newborns in Botswana. Methods: The Mpepu Study was a clinical trial that enrolled HEU newborns in Botswana to determine whether co-trimoxazole provided survival benefit. The Maikaelelo study was an observational study that enrolled HEU newborns in Botswana with telephone follow-up and no in-person visits. Mortality and morbidity were compared using survival analysis and treatment effects approaches. Cox-proportional hazards model were fit with time to death or time to first hospitalization and where the proportional hazards assumption was determined to be violated, interaction terms were generated between the study setting and time. The causal effect of study setting was obtained through the inverse probability weighted estimator. Results: In total, 4,010 infants were included; 1,306 were enrolled into Maikaelelo and 2,704 were enrolled into Mpepu. No significant differences in mortality were observed between the two settings (HR: 1.28, 95% CI: 0.76, 2.13), but RCT participants had a lower risk of hospitalization (HR: 0.72, 95% CI: 0.58, 0.89) that decreased with age. However, RCT participants had a higher risk of hospitalization within the first six months of life. The causal risk difference in hospitalizations attributable to the trial setting was -0.03 (95% CI: -0.05, -0.01). Conclusions: Children in an RCT with rigorous application of national standard of care guidelines experienced a significantly lower risk of hospitalization than children participating in an observational study that did not alter clinical care. Future research is needed to further investigate outcome disparities when real-world results fail to mirror those achieved in a clinical trial. Trial Registration: : The Mpepu Trial was funded by the U.S. National Institutes of Health (No. NCT01229761) and the Maikaelelo Study was funded primarily by the U.S. Centers for Disease Control and Prevention (32AI007433-21).