Correlated Chained Gaussian Processes for Datasets With Multiple Annotators

The labeling process within a supervised learning task is usually carried out by an expert, which provides the ground truth (gold standard) for each sample. However, in many real-world applications, we typically have access to annotations provided by crowds holding different and unknown expertise levels. Learning from crowds (LFC) intends to configure machine learning paradigms in the presence of multilabelers, residing on two key assumptions: the labeler’s performance does not depend on the input space, and independence among the annotators is imposed. Here, we propose the correlated chained Gaussian processes from the multiple annotators (CCGPMA) approach, which models each annotator’s performance as a function of the input space and exploits the correlations among experts. Experimental results associated with classification and regression tasks show that our CCGPMA performs better modeling of the labelers’ behavior, indicating that it consistently outperforms other state-of-the-art LFC approaches.