Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging.

Please see the Editorial Comment by Ian Amber discussing this article. Reported rates for performing recommended imaging in radiology reports are low. Bidirectional Encoder Representations from Transformers (BERT), a deep-learning model pre-trained to understand language context and ambiguity, has potential to identify recommendations for additional imaging (RAI) and thereby assist large-scale quality improvement efforts. To develop and externally validate an artificial intelligence (AI)-based model for identifying radiology reports containing RAI. This retrospective study was performed at a multisite health center. A total of 6300 radiology reports generated at one site from January 1, 2015 to June 31, 2021 were randomly selected and split by 4:1 ratio to training (n=5040) and test (n=1260) sets. A total of 1260 reports generated at the center's remaining sites (including academic and community hospitals) from April 1, 2022 to April 30, 2022 were randomly selected as an external validation group. Referring practitioners and radiologists of varying subspecialties manually reviewed report impressions for presence of RAI. A BERT-based technique for identifying RAI was developed using the training set. Performance of BERT-based model and a previously developed traditional machine-learning (TLM) model was assessed in the test set. Finally, performance was assessed in the external validation set. Model is publicly available: https://github.com/NooshinAbbasi/Recommendation-for-Additional-Imaging. Among 7419 unique patients, mean age was 58.8 years; 4133 were women, 3286 were men. Total of 10.0% of 7560 reports contained RAI. In test set, BERT-based model showed precision of 94%, recall of 98%, and F1 score of 96%; TML model showed precision of 69%, recall of 65%, and F1 score of 67%. In test set, accuracy was greater for BERT-based than TLM model (99% vs 93%, p<.001). In external validation set, BERT-based model showed precision of 99%, recall of 91%, F1 score of 95%, and accuracy of 99%. The BERT-based AI model accurately identified reports with RAI, outperforming TML model. High performance in the external validation set suggests the potential for other health systems to adapt the model without requiring institution-specific training. The model could potentially be applied for real-time EHR monitoring for RAI, or other improvement initiatives, to help ensure timely performance of clinically necessary recommended follow-up.