Challenges of Estimating Treatment Effects after a Positive Interim Analysis

Background This research investigates why a beneficial treatment effect reported at the first interim analysis (IA) may diminish at a subsequent analysis (SA). We examined three challenges in interpreting treatment effects from randomized clinical trials (RCTs) after the first positive IA: overestimation bias; non-proportional hazards; and heterogeneity in recruitment. We investigate how a penalized estimation method can address overestimation bias, and discuss additional factors to consider when interpreting positive IA results. Methods We identified oncology RCTs reporting positive results at the initial IA and a SA for event-free (EFS) and overall survival (OS). We modeled: (1) the hazard ratio at IA (HRIA) versus its timing as measured by the information fraction (IF; i.e., events at IA versus total events sought); and (2), the ratio of HRIA to HRSA (rHR) versus the IF. This was repeated for HRIA adjusted for overestimation bias. Examples of the other two challenges were sought. Results Amongst 71 RCTs, HRIA were positively associated with the IF (slope: EFS 0.83, 95% CI 0.44-1.22; OS 0.25, 95% CI 0.10 – 0.41). HRIA tended to exaggerate HRSA, and more so the lower the IF (slope rHR versus IF: EFS 0.10, 95% CI -0.22 – 0.42; OS 0.26, 95% CI 0.07 – 0.46). Adjusted HRIA did not exaggerate HRSA (slope rHR versus IF: EFS -0.14, 95% CI -0.67 – 0.39; OS 0.02, 95% CI -0.26 – 0.30). Examples of two other challenges are shown. Conclusion Overestimation bias, non-proportional hazards, and heterogeneity in recruitment and other important treatments should be considered when communicating estimates of treatment effects from positive IAs.