Using Kane’s Validity Theory to compare an Integrated and Single Skill Objective Structured Clinical Examination

Purpose The aim of this study was to compare the validity of an integrated Objective Structured Clinical Examination (OSCE) station assessing both oral and written components to an OSCE station assessing one single skill (oral only), both targeted at assessing taking a best possible medication history. Material and Methods A convergent mixed methods design that utilized the four inferences (scoring, generalization, extrapolation and implications) of Kane’s Validity Framework as a scaffold to integrate qualitative data (post OSCE reflections) and quantitative data (assessment grades and categories of medication errors) was applied. Results In 2022, 216 students completed the OSCE station with the oral component alone whilst in 2023, 254 students completed the integrated (oral and written) OSCE station. Students in 2023 performed significantly better, with a median score of 88% versus 80% in 2022 (p=0.002). There was a greater proportion of commission errors in the integrated assessment (20.4% vs 15.3%), but less omission errors (29.9% vs 31.8%) and patient profile errors (5.1% vs 69.4%). Student reflections revealed rushed conversations in the integrated assessment with a greater focus on written formatting, but an appreciation for the authenticity and structured format of the integrated OSCE compared to the single skill OSCE alone. Conclusion Students completing the integrated OSCE (with oral and written components) had less patient profile and medication omission errors than students who completed the oral only OSCE. Considering Kane’s Validity Framework, there was a more positive argument for the more authentic integrated OSCE in the inferences of extrapolation and implications.