Are patient satisfaction ratings reliable? Implications for holistic resident assessment

Purpose There is increasing interest in whether patient evaluations could serve as useful sources of formative assessment for surgical residents. This study aimed to understand the reliability of patient satisfaction ratings of surgical interns and senior residents and to estimate the number of patient ratings needed to support consistent and reproducible resident assessment. Methods English-speaking, surgery inpatients recovering from elective gastrointestinal and oncologic surgery completed a survey regarding their satisfaction with surgical resident care based on the Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey (S-CAHPS). Survey items on intern and senior resident satisfaction were analyzed using mixed-effects models to estimate variance components based on Generalizability Theory. Reliability projections were conducted to identify the number of surveys needed for score reproducibility. Results Ninety-one percent of approached patients participated ( n = 324/357, mean age = 62.2 years, 50.3% male). A total of 255 surveys were completed for 68 interns (3.8 assessments/ intern) and 269 surveys were completed for 43 senior residents (6.3/resident). Learner variance for surgical interns accounted for 7.1% and 1.6% of total variance for global and mean item-level behaviors, corresponding to Φ-coefficient reliability of 0.23 and 0.06, respectively. To achieve formative assessment reliability threshold of 0.70, reliability projections estimate needing 30 and 138 patient evaluations of surgical interns on the global and S-CAHPS items, respectively. For senior residents, learner variance accounted for < 0.01% variance resulting in inconsistent and inestimable reliability indices. Conclusions S-CAHPS based patient assessments may be most useful during the junior resident years given its higher variance among interns than senior residents. The significantly lower variability amongst senior residents may reflect a ceiling effect of high patient rating of senior resident care. This low variability suggests assessment tools beyond the S-CAHPS may be needed to differentiate meaningful and actionable behaviors of senior residents to inform patient-centered milestone achievement.