Towards Training Naive Participants For A Perceptual Annotation Task Designed For Experts

Technical Causes Analysis (P.TCA) is a method for identifying technical causes of sub-optimum speech transmission quality. Originally created as an expert procedure for the annotation of speech samples, its applicability to naive listener was also studied. Due to the low agreement of naive listener annotations, it was suggested that detailed training methods are necessary to lift naive annotations to an agreement level of experts. The aim of this work was to develop training methods for naive annotators. For this, two different training procedures were developed and tested in two separate annotation experiments. The results are analyzed and discussed regarding the effects of the trainings and their implications for the P.TCA annotation scheme. The outcome shows that these training methods did not meet the expectations for improving the inter-rater agreement of naive annotators. It is concluded that trainings of 15 to 20 minutes rather confuse naive annotators by conveying too much information in too little time, and that they are not sufficient to prepare naive annotators. It is argued that much more extensive training is needed to raise naive annotators to expert level, and that such a training must include both, in-depth introduction to the annotation process as well as detailed presentation and exercise regarding the P.TCA degradations.