Inter-rater Agreement for Visual Discrimination of Phasic and Tonic Electromyographic Activity in Sleep.

Study Objectives: The objective of this study was to determine the confidence of expert raters in discriminating phasic and tonic electromyographic (EMG) activity. We undertook this study because we suspected that even expert scorers may disagree on whether a given EMG segment contained phasic activity, tonic activity, or both. Methods: Six individuals holding either Fellowship status in the American Academy of Sleep Medicine or Board Certification in Sleep Medicine with at least 5 years experience in interpreting polysomnography visually examined 60 segments containing EMG activity. Raters determined their relative confidence that each segment contained phasic and tonic activity by noting whether they were highly certain or somewhat certain that the segment contained such activity or somewhat certain or highly certain that each segment did not contain such activity. Every segment was rated by every rater twice, once for phasic and once for tonic activity. Results: Substantial differences among raters existed in certainty regarding presence/absence of both phasic and tonic activity, although raters agreed on segments far above chance. Consensus was higher on certainty regarding presence of phasic, relative to tonic, activity. Conclusions: These findings indicate the limitations of visual analyses for discriminating abnormal muscle activity during sleep. Conversely, when expert judgments are combined with digitized measurements of EMG activity in sleep (e.g. REM atonia index), some allowance must be made for the unique contribution of visual analyses to such judgments, most notably for short duration EMG signals. These results may have relevance for polysomnographic interpretation in suspected synucleinopathies. Statement of Significance Recognition of features of electromyographic (EMG) activity during human sleep forms a cornerstone for the visual interpretation of polysomnography. Experienced raters examined segments of EMG activity and rated their confidence that the segment did nor did not contain phasic and tonic activity. Each segment was rated for both phasic and tonic activity. Although raters agreed far above chance in their certainty of the presence or absence of both phasic and tonic activity, they demonstrated much higher consensus over what constituted phasic activity. These results have implications for interpreting EMG signals in human sleep recordings. Determination of phasic muscle activity may be best appreciated visually, whereas distinguishing tonic muscle activity with confidence may necessitate quantitative, automated methods.