Retrospective large-scale evaluation of an AI system as an independent reader for double reading in breast cancer screening

Importance Screening mammography with two human readers increases cancer detection and lowers recall rates, but workforce shortages make double reading unsustainable in many countries. Artificial intelligence (AI) as an independent reader in double reading may support screening performance while improving cost-effectiveness. The clinical validation of AI requires large-scale, multi-vendor studies on unenriched cohorts. Objective To evaluate the performance of the Mia® AI system on data that the AI system would process in real-world deployments. Design A retrospective study simulating the impact of AI on an unenriched screening sample. Setting Seven European breast screening sites representing four centers: three from the UK and one in Hungary (HU), between 2009 and 2019. Participants The sample included 275,900 cases (177,882 participants) from seven screening sites, involving two countries and four hardware vendors from 2009 to 2019. Intervention Simulation of double reading using AI as an independent reader in breast cancer screening on historical data. Main Outcomes and Measures Performance was determined for standalone AI compared to the historical single reader and for simulated double reading with AI compared to historical double reading, assessing non-inferiority and superiority on relevant screening metrics using a non-inferiority margin of 10% relative difference and a one-sided alpha of 2.5% for both tests. Results Standalone AI detected 29.8% of missed interval cancers. When compared with historical double reading, double reading with AI showed non-inferiority for sensitivity and superiority for recall rate, specificity and positive predictive value. AI as an independent reader reduced the workload for the second human reader but increased the arbitration rate from 3.3% to 12.3%. Applying the AI system could have reduced the human reading time required by up to 44.8% and reduced the recall rate by a relative 7.7% (from 5.2% to 4.8%). Conclusions and Relevance Using the AI system as an independent reader maintains or improves the double reading standard of care, while substantially reducing the workload. Thus, it has the potential to provide operational and economic benefits. Trial Registration Registered on ISRCTN, study ID: ISRCTN18056078 ### Competing Interest Statement All authors have completed the ICMJE uniform disclosure form. Funding for the UK arm of the study was received from Innovate UK via an NHS England and Improvement, Office of Life Sciences (OLS) Wave 2 Test Bed Programme and a Medical Research Council (MRC) Biomedical Catalyst award. Authors affiliated with Kheiron Medical Technologies are paid employees. There are no financial relationships with any organizations that might have an interest in the submitted work in the previous three years and there are no other relationships or activities that could appear to have influenced the submitted work. ### Funding Statement The study was funded by Kheiron Medical Technologies. The following authors are employees of the company: Annie Y Ng, Galvin Khara, Georgia Fox, Ben Glocker, Edit Karpati, Tobias M Rijken, Joseph E Yearsley, Peter D Kecskemethy. The following were employees of Kheiron at the time of the study: Christopher C Austin, Andreas Heindl, Vignesh Venkataraman. Eva Ambrozay and Gabor Forrai are paid consultants of Kheiron. All other authors received no payment for this work. The UK arm of the study was supported by funding from Innovate UK via an NHS England and Improvement, Office of Life Sciences (OLS) Wave 2 Test Bed Programme and a Medical Research Council (MRC) Biomedical Catalyst award. ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The study had UK National Health Service Health Research Authority (REC reference 19/HRA/0376) and ETT-TUKEB Medical Research Council, Scientific and Research Ethics Committee, Hungary approval (reg no OGYEI/46651-4/2020). I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable. Yes The data for the current study are not publicly available. Due to reasonable privacy and ethical concerns, the imaging data cannot be distributed to researchers without ethical approval and research agreements with the original data providers.