Leveraging Crowdsourced Data for Creating Temporal Segmentation Ground Truths of Subjective Tasks
2013 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW)(2013)
Abstract
We present a new approach to the collection and labeling of ground truth data for annotation of temporal events in ad-hoc videos taken by active operators recording interactions and activities in the field. We present experimental data and related research from experimental psychology which indicate that the conventional methodology based on asking annotators to pick a single instance in time for an event boundary is both unnatural and has several undesirable effects. Our approach is based on allowing the annotators to choose event boundary intervals and modeling each annotators segmentations with mixtures of Gaussians. We use fuzzy measurements to determine an annotators quality and compute a segmentation likelihood function as a Gaussian Mixture of Models (GMMs) over all annotators and boundary intervals. Since the majority of evaluation methods require hard boundaries, we can extract these from the likelihood function as relevant local maxima. We show that given a small set of annotators, this GMM approach provides a more stable ground truth than conventional approaches including majority voting, and demonstrate the application of our approach on two segmentation problems.
MoreTranslated text
Key words
annotators quality,annotators segmentation,GMM approach,conventional approach,new approach,boundary interval,event boundary,event boundary interval,hard boundary,conventional methodology,Creating Temporal Segmentation Ground,Leveraging Crowdsourced Data,Subjective Tasks
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined