Evaluating the generalizability of deep learning image classification algorithms to detect middle ear disease using otoscopy

Abstract Purpose To evaluate the generalizability of artificial intelligence (AI)-otoscopy algorithms to identify middle ear disease using otoscopic images. Methods 1842 otoscopic images were collected from 3 independent sources: a) Van, Turkey, b) Santiago, Chile, and c) Ohio, USA. Diagnostic categories consisted of (i) normal or (ii) abnormal. Deep and transfer learning-based methods were used to develop models to evaluate internal and external performance, using area under the curve (AUC) estimates. A pooled assessment was performed by combining all cohorts together with 5-fold cross validation. Results AI-otoscopy algorithms achieved high internal performance (mean AUC: 0.95, 95%CI: 0.80–1.00). However, performance was reduced when tested on external otoscopic images not used for training (mean AUC: 0.76, 95%CI: 0.61–0.91). Overall, external performance was significantly lower than internal performance (mean difference in AUC: -0.19, mean standard error: 0.02, p≤0.04). Combining cohorts achieved a substantial pooled performance (AUC: 0.96, standard error: 0.01). Conclusion Internally applied AI-otoscopy algorithms performed well in identifying middle ear disease from otoscopy images. However, external performance was reduced when applied to new test cohorts. Further efforts are required to explore data augmentation and pre-processing techniques that might improve external performance and develop a robust, generalizable algorithm for real-world clinical applications.