Leveraging Crowdsourced Data for Creating Temporal Segmentation Ground Truths of Subjective Tasks

We present a new approach to the collection and labeling of ground truth data for annotation of temporal events in ad-hoc videos taken by active operators recording interactions and activities in the field. We present experimental data and related research from experimental psychology which indicate that the conventional methodology based on asking annotators to pick a single instance in time for an event boundary is both unnatural and has several undesirable effects. Our approach is based on allowing the annotators to choose event boundary intervals and modeling each annotators segmentations with mixtures of Gaussians. We use fuzzy measurements to determine an annotators quality and compute a segmentation likelihood function as a Gaussian Mixture of Models (GMMs) over all annotators and boundary intervals. Since the majority of evaluation methods require hard boundaries, we can extract these from the likelihood function as relevant local maxima. We show that given a small set of annotators, this GMM approach provides a more stable ground truth than conventional approaches including majority voting, and demonstrate the application of our approach on two segmentation problems.