Accuracy and generalizability of machine learning models for adolescent suicide prediction with longitudinal clinical records

Abstract Machine Learning (ML) models trained from real-world data (RWD) have demonstrated promise in predicting suicide attempts in adolescents. However, their cross-data performance and transportability for suicide prediction are largely unknown, hindering the clinical adoption of these ML models. We developed different ML suicide prediction models based on RWD collected in different contexts (inpatient, outpatient, etc.) with varying purposes (e.g., administrative claims and electronic health records), compare their cross-data performance, and evaluate their transportability. The data used was from the All-Payer Claims Database (APCD) and Hospital Inpatient Discharge Database (HIDD) in Connecticut as well as Electronic Health Records (EHR) data provided by Kansas Health Information Network (KHIN). From three datasets collected in different settings (inpatient, outpatient, etc.) and purposes (e.g., administrative claims and electronic health records), we included 285,320 patients among whom we identified 3389 (1.2%) suicide attempters. 66.0% of the suicide attempters were female. Different machine learning models (regularized logistic regression, gradient boosting machine, long-short term memory neural network) were evaluated on the local datasets and transported datasets. Significant and comparable decreases in the transfer performance compared to the local performance of all three ML models were observed, with the decline in performance reducing average AUC by up to 7.7%, reducing average sensitivity by up to 16%, and PPV by up to 2% at the 90% specificity level, and reducing sensitivity by up to 20% and PPV by up to 5% at the 95% specificity level. The similarity of behavior across these modeling approaches strengthens the validity of our results. The commonality and heterogeneity of predictors learned across populations were compared. These results indicate that no matter how well ML suicide models performed with their source data, their performance when transporting these models to new datasets is limited. However, the transported models did identify additional new cases. Our analyses could facilitate the development of suicide prediction models with better performance and generalizability.