Harnessing the Wisdom of the Confident Crowd in Medical Image Decision-Making

Improving the accuracy of medical image interpretation is critical to improving the diagnosis of many diseases. Using both novices (undergraduates) and experts (medical professionals), we investigated methods for improving the accuracy of a single decision maker and a group of decision makers by aggregating repeated decisions in different ways. Participants made classification decisions (cancerous vs. noncancerous) and confidence judgments on a series of cell images, viewing and classifying each image twice. We first examined whether it is possible to improve individual-level performance by using the maximum confidence slating (MCS) algorithm (Koriat, 2012b), which leverages metacognitive ability by using the most confident response for an image as the "final response." We find MCS improves individual classification accuracy for both novices and experts. Building on these results, we show that aggregation algorithms based on confidence weighting scale to larger groups of participants, dramatically improving diagnostic accuracy, with the performance of groups of novices reaching that of individual experts. In sum, we find that repeated decision-making and confidence weighting can be a valuable way to improve accuracy in medical image decision-making and that these techniques can be used in conjunction with each other.