External validation of the HACOR score and ROX index for predicting treatment failure in patients with coronavirus disease 2019 pneumonia managed on high-flow nasal cannula therapy: a multicenter retrospective observational study in Japan

Background The HACOR score for predicting treatment failure includes vital signs and acid–base balance factors, whereas the ROX index only considers the respiratory rate, oxygen saturation, and fraction of inspired oxygen (FiO 2 ). We aimed to externally validate the HACOR score and ROX index for predicting treatment failure in patients with coronavirus disease 2019 (COVID-19) on high-flow nasal cannula (HFNC) therapy in Japan. Methods This retrospective, observational, multicenter study included patients, aged ≥ 18 years, diagnosed with COVID-19 and treated with HFNC therapy between January 16, 2020, and March 31, 2022. The HACOR score and ROX index were calculated at 2, 6, 12, 24, and 48 h after stating HFNC therapy. The primary outcome was treatment failure (requirement for intubation or occurrence of death within 7 days). We calculated the area under the receiver operating characteristic curve (AUROC) and assessed the diagnostic performance of these indicators. The 2-h time-point prediction was considered the primary analysis and that of other time-points as the secondary analysis. We also assessed 2-h time-point sensitivity and specificity using previously reported cutoff values (HACOR score > 5, ROX index < 2.85). Results We analyzed 300 patients from 9 institutions (median age, 60 years; median SpO 2 /FiO 2 ratio at the start of HFNC therapy, 121). Within 7 days of HFNC therapy, treatment failure occurred in 127 (42%) patients. The HACOR score and ROX index at the 2-h time-point exhibited AUROC discrimination values of 0.63 and 0.57 ( P = 0.24), respectively. These values varied with temporal changes—0.58 and 0.62 at 6 h, 0.70 and 0.68 at 12 h, 0.68 and 0.69 at 24 h, and 0.75 and 0.75 at 48 h, respectively. The 2-h time-point sensitivity and specificity were 18% and 91% for the HACOR score, respectively, and 3% and 100% for the ROX index, respectively. Visual calibration assessment revealed well calibrated HACOR score, but not ROX index. Conclusions In COVID-19 patients receiving HFNC therapy in Japan, the predictive performance of the HACOR score and ROX index at the 2-h time-point may be inadequate. Furthermore, clinicians should be mindful of time-point scores owing to the variation of the models’ predictive performance with the time-point. Trial registration UMIN (registration number: UMIN000050024, January 13, 2023)