The Use of Learning Analytics to Enable Detection of Underperforming Trainees An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data

Objective:This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets. Summary Background Data:ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation. Methods:National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings. Results:A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies (P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs. Conclusion:The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.