Assessing Potential Bias from Metadata while Labeling Retinal Images Fundus Photographs for Diabetic Retinopathy: Preliminary Experience in the Multimodal Database of Retinal Images in Africa (MoDRIA) (Preprint)

BACKGROUND Labeling color fundus photos (CFP) is an important step in the development of artificial intelligence (AI) screening algorithms for the detection of diabetic retinopathy. The International Classification of Diabetic Retinopathy (ICDR) is used to assign labels to CFP, plus the presence or absence of macular edema. Images can be grouped as referrable or non-referrable for treatment. There is little guidance in the literature about how to collect and use clinical metadata as a part of the CFP labeling process. OBJECTIVE To improve the quality of the Multimodal Database of Retinal Images in Africa (MoDRIA) by determining whether the availability of clinical metadata during the image labeling process influences the accuracy, sensitivity, and specificity of image labels. METHODS This is a crossover assessment with 2 groups and 2 phases. Each group had 10 randomly assigned labelers who provided an ICDR score and presence or absence of macular edema for each of 50 CRF in a test image with and without metadata. Specificity and sensitivity of referable retinopathy was based on ICDR scores, and macular edema calculated using 2-sided T-test. Comparison with and without metadata for each participant was calculated using the signed rank test. Statistical significance was set at P<0.05. RESULTS The sensitivity for identifying referrable diabetic retinopathy with metadata was 92.8% (95% CI: 87.6-98.0) compared with 93.3% (95% CI: 87.6-98.9) without metadata, and the specificity was 84.9% (95% CI: 75.1-94.6) with metadata compared with 88.2% (95% CI: 79.5-96.8) without metadata. The sensitivity for identifying the presence of macular edema was 64.3% (95% CI: 57.6-71.0) with metadata, compared with 63.1% (95% CI: 53.4-73.0) without metadata, and the specificity was 86.5% (95% CI: 81.4-91.5) with metadata compared with 87.7% (95% CI: 83.9-91.5) without metadata. Sensitivity and specificity of ICDR score and presence or absence of ME were also calculated for the 20 individual labelers with and without metadata. No findings were statistically significant. CONCLUSIONS In this quality improvement project, clinical metadata availability did not influence labeling quality. Additional studies are needed to understand the potential implications of the process and components of labeling with and without metadata more thoroughly with regards to accuracy and bias. These issues have far reaching implications given the rapidly expanding use of AI with clinical images, including on the African continent.