Contrast between traditional and machine learning algorithms based on a urine culture predictive model: a multicenter retrospective study in patients with urinary calculi

Background: Quick and accurate identification of urinary calculi patients with positive urinary cultures is critical to the choice of the treatment strategy. Predictive models based on machine learning algorithms provide a new way to solve this problem. This study aims to determine the predictive value of machine learning algorithms using a urine culture predictive model based on patients with urinary calculi. Methods: Data were collected from four clinical centers in the period of June 2016, to May 2019. 2,054 cases were included in the study. The dataset was randomly split into ratios of 5:5, 6:4, and 7:3 for model construction and validation. Predictive models of urine culture outcomes were constructed and validated by logistic regression, random forest, adaboost, and gradient boosting decision tree (GBDT) models. Each ratio's construction and verification were repeated five times independently for cross-validation. The Matthews correlation coefficient (MMC), F1-score, receiver operating characteristic (ROC) curve with the area under curve (AUC) was used to evaluate the performance of each prediction model. The additive net reclassification index (NRI) and absolute NRI were used to assess the predictive capabilities of the models. Results: Four prediction models of urinary culture results in patients with urinary calculi were constructed. The mean AUCs of the logistic regression, random forest, adaboost, and GBDT models were 0.761 (95% CI: 0.753-0.770), 0.790 (95% CI: 0.782-0.798), 0.779 (95% CI: 0.766-0.791), and 0.831 (95% CI: 0.823- 0.840), respectively. Moreover, the average MMC and F1-score of GBDT model was 0.460 and 0.588, which was improved compared to logistic regression model of 0.335 and 0.501. The additive NRI and absolute NRI of the GBDT and logistic regression models were 0.124 (95% CI: 0.106-0.142) and 0.065 (95% CI: 0.060-0.069), respectively. Conclusions: Our results indicate that machine learning algorithms may be useful tools for urine culture outcome prediction in patients with urinary calculi because they exhibit superior performance compared with the logistic regression model.