Does balancing site characteristics result in balanced population characteristics in a cluster-randomized controlled trial?

Background Intervention trials with nested designs seek to balance sites randomized regarding key site characteristics. Among the goals of such site-level balancing is to accrue patient-level equivalence among treatment arms. We investigated patient-level equivalence in a cluster randomized controlled trial, which balanced study waves on site-level characteristics. Methods The Behavioral Health Interdisciplinary Program—Collaborative Chronic Care Model project utilized a stepped wedge design to stagger implementation of an evidence-based team-oriented mental health patient management system at 9 Veteran Affairs Medical Centers. Study sites were balanced on eight site-level characteristics over time (3 balanced waves [consecutive time periods] with 3 sites per wave) to minimize trend. Sites were balanced on selected site-level characteristics but not on patient-level variables. We explored internal differences in patient demographics across the three study waves. Eligible patients had at least two visits to a participating mental health clinic in the prior year and did not have a diagnosis of dementia (n = 5,596). Results We found modest but statistically significant inter-site differences in age, marital status, ethnicity, service-related disability, mental health hospitalizations, and selected diagnoses by study wave. Although many of the differences in patient demographics by study wave were statistically significant, only a few results were practically meaningful as measured by effect size. A bipolar diagnosis (49.0%, 21.0%, 17.0% in waves 1–3, respectively; Cramer’s V = 0.3124) and Hispanic ethnicity (2.9%, 29.6%, 2.0% in waves 1–3, respectively; Cramer’s V = 0.3949) resulted in differences that were considered a ‘moderate’ effect size. The number of patient characteristics that were both statistically and meaningfully different by study wave among all possible site assignments was comparable to the 34 most balanced site assignments identified in our balancing algorithm. Conclusions Using a balancing algorithm to reduce imbalance among site characteristics across time periods did not appear to negatively affect the balance of patient characteristics across sites over time. A site-level balancing algorithm that includes characteristics with a direct relationship to relevant patient-level factors may improve the overall balance across key elements of the study, and aide in the interpretation of results.