Inter-rater disagreement in manual scoring of intensive care unit sleep data

Abstract Objective: Severe sleep disruption is common among intensive care unit (ICU) patients. However, the applicability of standard sleep scoring guidelines by the American Academy of Sleep Medicine (AASM) has been questioned, with most polysomnography (PSG) studies in critically ill patients reporting difficulties in setting up and processing and scoring the recordings. The present study explores human inter-rater agreement in sleep stage scoring following the AASM guidelines, within a heterogenous ICU patient cohort. Results: Two human experts independently scored a total of 51,454 epochs in 20 PSG recordings acquired at the ICU. Epoch-per-epoch comparison of scored stages revealed a Cohen’s κ coefficient of agreement of 0.36 for standard 5-stage scoring. Highest agreement occurred in Wake (κ = 0.46), while REM showed the lowest (κ = 0.12). Significant correlations were found between inter-rater agreement, and Simplified Acute Physiology Score (SAPS II, r=-0.506, p=0.038), and 12-month mortality (r=-0.524, p=0.031). Comparison with similar studies underscore challenges in applying AASM criteria to ICU patients. Despite accounting for artifacts, disparities persisted, emphasizing the need for a nuanced exploration of factors influencing scoring inconsistencies in critically ill patients.