External Validation of a Neural Network Model in Aneurysmal Subarachnoid Hemorrhage: A Comparison With Conventional Logistic Regression Models

BACKGROUND: Interest in machine learning (ML)-based predictive modeling has led to the development of models predicting outcomes after aneurysmal subarachnoid hemorrhage (aSAH), including the Nijmegen acute subarachnoid hemorrhage calculator (Nutshell). Generalizability of such models to external data remains unclear.OBJECTIVE: To externally validate the performance of the Nutshell tool while comparing it with the conventional Subarachnoid Hemorrhage International Trialists (SAHIT) models and to review the ML literature on outcome prediction after aSAH and aneurysm treatment.METHODS: A prospectively maintained database of patients with aSAH presenting consecutively to our institution in the 2013 to 2018 period was used. The web-based Nutshell and SAHIT calculators were used to derive the risks of poor long-term (12-18 months) outcomes and 30-day mortality. Discrimination was evaluated using the area under the curve (AUC), and calibration was investigated using calibration plots. The literature on relevant ML models was surveyed for a synopsis.RESULTS: In 269 patients with aSAH, the SAHIT models outperformed the Nutshell tool (AUC: 0.786 vs 0.689, P = .025) in predicting long-term functional outcomes. A logistic regression model of the Nutshell variables derived from our data achieved adequate discrimination (AUC = 0.759) of poor outcomes. The SAHIT models outperformed the Nutshell tool in predicting 30-day mortality (AUC: 0.810 vs 0.636, P < .001). Calibration properties were more favorable for the SAHIT models. Most published aneurysm-related ML-based outcome models lack external validation and usable testing platforms.CONCLUSION: The Nutshell tool demonstrated limited performance on external validation in comparison with the SAHIT models. External validation and the dissemination of testing platforms for ML models must be emphasized.