Validation and calibration of machine‐learning predictive models aimed to identify dementia‐related neuropsychiatric symptoms on real‐world data (RWD)

Background 8% of currently under evaluation new drugs for Alzheimer’s disease are intended to treat neuropsychiatric symptoms (NPS). But, the lack of their presence in codded format in clinical records registries determines that their population impact is underestimated. Validation studies are required in order to systematically use real‐world data (RWD) as a source of epidemiological information for dementia. The objective of this study was to build and validate predictive models to identify the presence of both behavioral or psychotic and depressive symptoms in dementia‐diagnosed patients from administrative and clinical databases. Method We searched the Electronic Health Recordings (EHR) of 4,003 patients with dementia looking for mood and psychotic symptoms. First, codded diagnoses, drugs prescriptions and other clinical variables were used to identify the dementia cases in the Basque Service corporate database. Second, Random Forests algorithms were applied to build predictive models by using the training sample (N=3,003) to select the best models separately for mood and behavioral symptoms. Third, we measured calibration and discrimination in the remaining testing sample (N= 1000) separately for each model. Result Neuropsychiatric Symptoms (NPS) were identified in 58% of the sample. The model predicting psychotic symptoms achieved 80% of AUC when depressive model only achieved 74%. Kappa index and accuracy also showed better discrimination in the psychotic symptoms model. As displayed by the calibration plot, the models made worse predictions when the probability of the cases was lower than 25%. The most relevant variables in the psychotic symptoms model were the highest sedation level achieved followed by the use of quetiapine and risperidone and the number of different antipsychotic treatments. In the depressive symptoms model, the most relevant variable is the number of antidepressant treatments followed by the use of escitalopram, sedation level and age. Calibration was poorer when the probability was low. Conclusion Given the good performance, the resulting models can be applied to estimate the prevalence of NPS in population databases (RWD). Our work showed how machine learning tools can be used to predict complex clinical conditions like NPS and possess the capacity to convert RWD into smart data.