On Subset Selection of Multiple Humans To Improve Human-AI Team Accuracy

There are several classification tasks where neither the human nor the model is perfectly accurate. Some recent works, therefore, focus on the Human-AI team model, where the AI model's probabilistic output is combined with the human-predicted class label. The combined decision is shown to consistently outperform the model's or human's accuracy alone. All the previous works, however, restrict to the setting where they consider a single human to combine with the AI model. Motivated by the crowdsourcing literature, which combines labels from multiple humans, we show that combining multiple human labels with the model's probabilistic output can lead to significant improvement in accuracy. This paper further shows that while combining multiple humans helps, a naive combination of humans with AI model can lead to poor accuracy. Hence, there is a strong need for an intelligent strategy to select a subset of humans and combine their labels. To this end, we present an approach to merge the predicted labels from multiple humans with the model's probabilistic output. We then provide an efficient algorithm to find the optimal subset of humans whose combined labels offer the most accurate output. Finally, we empirically demonstrate that the combined model outperforms the AI model or any human alone in terms of accuracy. Besides this, our subset selection algorithm and combination method outperforms the single human model and other naïve combination techniques.