OSATS Tool for Pediatric Rapid Sequence Intubation

Abstract Introduction Being procedurally competent to perform pediatric rapid sequence intubation (pRSI) is an ACGME requirement for emergency medicine residents. There are few assessment tools to assess this technique with adequate validity evidence. An objective structured assessment of technical skills for pRSI (OSATS-pRSI) was developed and validity evidence was obtained in four of five areas of validity: content, relation to other variables, and inter-rater reliability. Methods This assessment tool was designed to be used to evaluate learner's performance during pPRSI using a standard respiratory distress scenario and a pediatric high-fidelity simulator (Laerdal SimBaby). The scenario was a 7-month old infant who presented to the Emergency Department in respiratory distress. A framework of a previously established objective structured assessment of technical skills (OSATS) tool was modified for pRSI by a panel of pediatric and general emergency medicine physicians. Emergency medicine residents (postgraduate year 1-4) were prospectively enrolled in a pRSI simulation scenario and evaluated by two raters using the modified tool. Results Sessions were videotaped and reviewed by the same raters at least 4 months later. Raters were blinded to their initial rating. Interrater agreement was determined by using the Krippendorff generalized concordance method. Overall interrater agreement for live review was 0.75 (95% CI = 0.72 — 0.78) and for video was 0.79 (95% CI = 0.73 — 0.82). Live review was significantly superior to video review in only one of the OSATS domains (preparation) and was equivalent in the other domains. Intrarater agreement between the live and video evaluation was very good, greater than 0.75 for all raters, with a mean of 0.81 (95% CI = 0.76 — 0.85). Two domains of the modified OSATS instrument (preintubation process and intubation technique) were able to discriminate trainees by level of training and neonatal experience. Discussion The modified OSATS assessment tool demonstrates validity evidence in content, inter- and intra-rater reliability, and relation to other variables. There are limited number of assessment tools for pRSI with validity evidence. This tool has demonstrated validity evidence in content, response process, and inter- and intra-rater reliability. This tool was successfully piloted by 45 emergency medicine residents and showed increased performance in two domains with increasing PGY level.