Addressing the need to develop critical thinking skills in the new competency-based medical education post graduate curriculum in pathology: Experience-sharing of the process of development and validation of script concordance test

Background and Aim: Ability to integrate and apply knowledge to clinical practice is recognized as a core competency under the new CBME curriculum and assessment of critical thinking skills is an essential component. Script Concordance Test (SCT) is recognised as an appropriate assessment tool for measuring critical thinking. Since in India, SCTs for testing critical thinking in the Postgraduate curriculum in Pathology, have not been developed, we worked on developing such test in the setting of a busy referral hospital attached to Medical College in Western India. The study aimed at 1. Piloting the development of script concordance test to test critical thinking ability and clinical reasoning skills in haematology. 2. Experience sharing of the process of development and validation of the test to enable others to replicate for other topics in pathology. Materials and Methods: After obtaining Ethical clearance, SCT for the topic Coagulation was constructed by following the guidelines from AMEE Guide 75. The experts' reactions and their satisfaction about the test were obtained. Pilot testing was done with post graduate students after informed consent. The students' reactions and satisfaction were obtained after taking the test. The analysis was performed using the excel workbook available from a web link maintained by the SCT creators. Item analysis and group comparison was also done. Results: Analysis performed with 22 vignettes having 66 items revealed a statistically significant difference between the scores of the experts and the students. Internal consistency reliability (Cronbach's α) was 0.86 whereas Effect size (Cohen's d) was 1.9 with 34% overlap in the items and superiority probability of 91%. Item analysis classified 23 items as “Bad”, 11 items as “Fair” and 32 items as “Good” based on the “item-total correlation” statistic. Re-analysis after removing the bad items, increased Cronbach's α to 0.91 and Cohen's d to 3.9; whereas overlap decreased to 5% and probability of superiority reached 100%. Conclusion: Development of a reliable and valid SCT for coagulation was a complex but rewarding process. The analysis of the test required inputs from the statistical experts. Using AMEE Guide 75 and our in-detail description of the process of development, optimisation and validation of script concordance test will enable faculty in many specialities to develop and include it for use in their respective fields.