Reply to Adelman et al.

To the Editor—We appreciate Adelman and colleagues’ [1] thoughtful response to our study [2]. We agree that sepsis diagnosis is complex and that it is important to consider whether our adjudication protocol yielded reproducible and accurate results. One potential limitation noted is that the final presence of infection was adjudicated by more than 1 person for only a subset of our cohort. However, interrater agreement was high and comparable to past studies [3, 4]—κ = 0.69 (95% confidence interval [CI]: .60–.78) for the binary determination of infection presence and κ = 0.83 (95% CI: .80–.86) for infection presence and source [5] — and there was not a systematic pattern of between-rater disagreement (Figure 1). As such, it seems unlikely that additional two-reviewer adjudications would have substantially altered our findings. We employed a panel of trained research assistants and medical students for most chart adjudication and validation. While this pragmatic strategy allowed us to evaluate a cohort substantially larger than the cited studies that used multiphysician adjudication (n = 211 [6], n = 447 [7], and n = 2579 [8]), we concur that the use of nonclinician adjudicators is a possible limitation of our study. However, as noted by Adelman and colleagues, even physician adjudication is not perfect, with the determination of the presence of sepsis varying substantially between physicians. In one study, clinicians who were given a series of case vignettes but no structured adjudication criteria exhibited poor agreement regarding the presence or absence of sepsis (κ only 0.18) [9]. By contrast, a study that used structured adjudication criteria had interrater agreement similar to the agreement in our study (κ = 0.79) between a two-physician panel with all available discharge information and a “gold standard” external 3-physician panel [3]. Taken together, these data suggest that expert judgment is insufficient for reliable and reproducible sepsis adjudication, while reproducible adjudication criteria like those used in our study are critical. It is also worth considering whether physicians can bring systematic biases to the adjudication task. A recent study found that infectious diseases specialists had higher thresholds for recognizing and treating infection than critical care and emergency medicine physicians [10], suggesting that the training background of physician adjudicators could influence study findings.