Automatic web accessibility metrics: Where we are and where we can go

Abstract

a b s t r a c tThe fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.

Assessment of Web Content Accessibility Levels in Spanish Official Online Education Environments

“…After admitting the limitations of automated analysis, several authors reached the conclusion that if a certain level is not obtained with an automated tool, that level cannot possibly be reached manually (Freire, Bittar, & Fortes, 2008;Vigo & Brajnik, 2011;Buhler, Heck, Perlick, Nietzio, & Ulltveir-Moe, 2006). The results obtained in our study match the uniformly negative trends for diversity-based design of other research studies.…”

Redesigning websites for older adults: a case study

“…WCAG 2.0 includes four principles ((1) Perceivable, (2) Operable, (3)Understandable, and (4) Robust) and a total of 12 guidelines. The specification has a highly technical orientation and requires good knowledge of web technologies for its comprehension and application (Koutsabasis et al 2010), while a good portion of the proposed guidelines can be technically validated (Vigo and Bajnik 2011). Web accessibility guidelines are being employed at the late stages of website development and tend to reveal complementary design and implementation issues to those emerging from usability studies.…”

Investigating the Potential of a Dashboard for Monitoring U.S. Federal Website Accessibility

“…Hanson and Richards [4] chose the percentage of violations of all sites in their sample. Others have attempted to come up with metrics that are computed, for instance the Web Accessibility Barrier score is a weighted average involving the number of accessibility barriers and barriers are prioritized based on how the violation is ranked within the WCAG, and the Web Accessibility Quantitative Metric prioritizes barriers that can be evaluated by automated tool, over those that require some human checking [13]. Vigo and Brajnik clearly state the challenges: "we are faced with a conundrum: on the one side we have quick reliable but potentially invalid ways to measure accessibility, on the other side we have expensive methods to evaluate accessibility, subject to a smaller degree of invalidity and unreliability…" [13, p.2].…”