The impact of pre-testing on post-test bronchoscopy competence in a simulation setting.

Background and aim: Bronchoscopy simulation studies often use a pre-test post-test design. However, a drawback of this design is a possible testing effect, resulting in an overestimation of intervention effects. We evaluated the effect of pre-testing on post-test bronchoscopy simulation performance. Methods: Undergraduate medical students without any bronchoscopy experience were recruited for this study through flyers on campus. Twenty students participated between November 2021 and October 2022. Participants performed 2 simulator tasks in the morning (the pre-test). After a 4-6 hour break, participants performed the same 2 tasks (the post-test). The tasks comprised “Task 1” (assessing navigational skills) and “Task 3” (assessing navigational skills in a realistic anatomical environment). Outcome measures were simulator metrics for Task 1 and validated assessment tool scores (adapted from Konge, L. et al. Respiration 2012; 83:53-60) and time for Task 3. The tool consisted of 7 parameters leading to an overall score on a 5-point scale (1 representing the worst and 5 the best performance). Results: Results showed on average a 40% reduction in wall contacts for Task 1 (P<.001) and a reduction in time needed for Task 1 (57 ± 15 seconds pre versus 38 ± 12 seconds post, P<.001) and Task 3 (5.3 ± 1.3 minutes pre versus 4.4 ± 1.0 minutes post, P<.001). Overall assessment tool scores did not improve (pre 2.0[1.0], post 2.0[0.0], P=0.4). Conclusion: A significant testing effect was observed for navigational skills in general and in a realistic anatomical environment. Future pre-post-test bronchoscopy simulation studies should correct their effect sizes taking into account the size of this testing effect.