Capturing Surgical Data: Comparing a Quality Improvement Registry to Natural Language Processing and Manual Chart Review

Introduction Collecting accurate operative details remains a limitation of surgical research. Surgeon-entered data in clinical registries offers one solution, but natural language processing (NLP) has emerged as a modality for automating manual chart review (MCR). This study aims to compare the accuracy and efficiency of NLP and MCR with a surgeon-entered, prospective registry data in determining the rate of gross bile spillage (GBS) during cholecystectomy. Methods Bile spillage rates were abstracted from an institutional, surgeon-entered clinical registry from July 2018 to January 2019. These rates were compared to those documented in the electronic medical record (EMR) using NLP and MCR to determine the sensitivity, specificity, and efficiency of each approach. Results Of the 782 registry entries, 191 cases (24.4%) had surgeon-reported bile spillage. MCR identified bile spillage in 121 cases (15.6%); however, bile spillage information was either missing or ambiguous in 454 cases (58.1%). NLP identified 99 cases (12.7%) of bile spillage. Data abstraction times for the registry, NLP, and MCR were 3 min, 5 min, and 12 h, respectively. When compared to the registry, MCR was 45% sensitive and 94% specific, while NLP was 27.2% sensitive and 92% specific for detecting bile spillage. These differences were significant ( X 2 = 19.446, P = < 0.001). Conclusion Operative details, such as GBS, may not be abstracted by NLP or MCR if not clearly documented in the EMR. Clinical registries capture operative details, but they rely on surgeons to input the data.