The use of WCAG and automatic tools by computer science students: a case study evaluating MOOC accessibility

Web Content Accessibility Guidelines (WCAG) have been the de facto standard for Web accessibility evaluation for more than two decades and therefore have been introduced into legislation and university curriculum in Computer Science. At The National Distance Education University (UNED) in Spain, we have been teaching the guidelines for the last 15 years but learning how to apply WCAG criteria is complex. In this paper, we present the results of the analysis of students' performance in applying accessibility heuristic evaluation of an online resource (a Massive Open Online Course - MOOC) using WCAG. The experiment was carried out over two academic years to evaluate how accurate and easy it is to understand and use WCAG criteria by trained students as well as their perceptions of usefulness to evaluate accessibility barriers using automatic tools in combination with manual evaluation. Results from the study show that errors identified are aligned with accessibility evaluation literature: 65% of success criteria in WCAG do not reach 80% of agreement among raters which confirms the complexity of WCAG conformance. In total 62 (86%) criteria are marked as not being correctly addressed by automatic tools with an overlap of those showing false positives, and 25 criteria (34%) are indicated as difficult to evaluate manually. While all areas where raters disagree are potential opportunities for WCAG improvement, this research reinforces that WCAG evaluations are complex and difficult even with current automatic tools, and that the possible solutions for the way forward are: (1) a well-defined evaluation protocol including a combination of automatic tools and manual evaluations; (2) better training and professional development opportunities.