11. Evaluating the ACAPS Standardized Letter of Recommendation for Application to Plastic Surgery Residency: Usefulness and Patterns

The selection of qualified applicants who will become successful physicians is an important and challenging task for residency programs of all specialties. In addition, recent changes in medical training and education (e.g., pass/fail curricula, clerkships and board examinations, along with limited subinternships) have made it more difficult to determine objective criteria to differentiate among applicants. As these distinguishing applicant factors continue to decrease, the remaining components of the application for residency have become more important.1 The letter of recommendation is one of the most important factors considered by many programs,2 and in this changing environment, it will become even more emphasized. The American Council of Academic Plastic Surgeons (ACAPS) created a standardized letter of recommendation form in 2012. Many plastic surgery program directors believed that narrative letters of recommendation “did not offer a realistic way to compare applicants” and desired a way to increase the objectivity of these letters.3 However, since its introduction, there have been no published studies evaluating its efficacy. We aimed to identify patterns in the ACAPS standardized letter of recommendation form to help with interpretation for applicant selection as well as suggest potential areas for improvement. All letters of recommendation submitted to our institution for application to residency over a 4-year span (2017 through 2020; 1591 total letters) were evaluated. Overall, 82.3 percent of letters received included the ACAPS standardized letter of recommendation form. Our study showed significant rank inflation, with almost 75 percent of applicants ranked between one and five and almost 95 percent being ranked as being in the top 10 (Table 1). This results in a skewing of the bell curve to the left and is probably not an accurate representation of the overall applicant pool. Home program letters were much more likely to rank applicants as number one compared with away programs, which were more likely to rank applicants in the five to 10 and 10 to 20 categories (Table 2). There were no overt differences found in rank between men and women. Program director rankings were not statistically different from those of other letter writers. Table 1. - Distribution of Overall Rank Overall Rank No. of Letters Percentage 1 326 25.7 2 to 5 621 48.9 5 to 10 256 20.2 10 to 20 65 5.1 Would not rank 1 0.1 Total 1269 100 Table 2. - Differences in Home versus Away Program Rankings Overall Rank Home Letters Away Letters p 1 35.4% 13.0% <0.001 2 to 5 47.7% 50.5% 0.34 5 to 10 13.7% 28.6% <0.001 10 to 20 3.2% 7.7% <0.001 Would not rank 0.0% 0.2% 0.43 Total 721 548 The overall rank selection is often used as a summative statement of the standardized letter of recommendation, and many evaluators anchor on this portion of the evaluation. However, our results show that it is not as reliable as one would hope. This may be related to the selection choices provided. Reghunathan et al.4 identified several areas for improvement to the ACAPS standardized letter of recommendation form by surveying ACAPS faculty. Most notable was the movement toward use of percentiles for the attributes section. Perhaps moving toward a percentile format for overall rank would provide more evenly distributed and comparable data. The standardized letter of recommendation is highly used in the evaluation of applicants to plastic surgery residency and analyses into its patterns are important. Caution must be used when interpreting the overall ranking as the distribution is heavily skewed and strong biases exist between home and away program letters. Understanding trends in the standardized letter of recommendation form will be vital as it gains importance in the evaluation of applicants. Further investigation on ways to improve the implementation and design of the standardized letter of recommendation form is needed. DISCLOSURE The authors have no financial interest to declare in relation to the content of this article.