Towards Automated Web Accessibility Evaluation: A Comparative Study

Abstract

Abstract-With each passing day, the Web is becoming increasingly important in our lives. Hence, the need of making it more accessible to everyone, especially for the disabled and elderly spurred a great interest in automated tools, the total registered number of which has been continuously increasing and reached from forty-five software bids in 2014 to ninety-three in 2017. The purpose of this empirical research is to assess and compare eight popular and free online automated Web accessibility evaluation tools (AWAETs) such as AChecker, Cynthia Says, EIII Checker, MAUVE, SortSite, TAW, Tenon and WAVE with regard to the WCAG 2.0 conformance. As a result, significant differences were observed in terms of tool's coverage (a maximum of 32.4%), completeness (ranges between 10% and 59%), correctness (an average of 70.7%), specificity (reaches 32%), inter-reliability (lies between 1.56% and 18.32%) and intra-reliability (the acceptable score), validity, efficiency and capacity. These eight criteria can help to determine a new role played by modern AWAETs as dependent methods in Web accessibility evaluation. Moreover, consequences of relying on AWAETs alone are quantified and concluded that applying such approaches is a great mistake since subjective and less frequent objective success criteria (SC) failed to be automated. However, using a good combination of AWAETs is highly recommended as overall results in all the mentioned quality criteria are maximized and tools could definitely validate and complete each other. Ultimately, integrating automated methods with the others is ideal and preferably at an early stage of the website development life cycle. The study also provides potential accessibility barriers that make websites inaccessible, challenges AWAETs are currently facing, nineteen pros and fourteen cons and fifteen improvement recommendations for the existing and next generation of AWAETs. Fundamentally, achieving the objectives of this study was possible due to the elaboration and implementation of a new five-phased methodology named as -5PhM-for-AWAEMs‖ for successful selection, evaluation and/or comparison of AWAEMs. In addition to providing detailed descriptions of the estimation process, this methodology represents eleven key criteria for effective selection of suitable AWAEMs and necessary numbers of web pages and expert evaluators for acceptable, normal or ideal assessment.

Internet Accessibility for Blind and Visually-Impaired Users: An Evaluation of Official U.S. State and Territory COVID-19 Websites

“…The checkers have been compared, validated, and described in detail in previous research (Abduganiev, 2017;Ismail et al, 2018). It is well-established that multiple checkers should be used when researching accessibility of webpages (Timbi-Sisalima et al, 2016).…”

Introdução à Acessibilidade na Web: do Conceito à Prática

“…Mas, apesar dessas limitações, sua utilização ainda é válida e pode ajudar na melhoria do nível de acessibilidade das páginas avaliadas. É aconselhável utilizar mais de uma ferramenta para que elas se complementem (uma ferramenta pode cobrir um ponto falho da outra) [Abduganiev 2017].…”

“…Por causa de sua importância no processo de criação dos web sites, advinda da velocidade proporcionada na identificação de erros de acessibilidade, os avaliadores automáticos continuam a ser estudados pela comunidade científica, e novas ferramentas continuam a ser desenvolvidas [Abduganiev 2017]. Há ainda ferramentas desenvolvidas com recursos de inteligência artificial para que cada vez mais verificações possam ser feitas automaticamente [Alsaeedi 2020].…”

Requirements for Large Scale Web Accessibility Evaluation

“…Overall, a solution to Web accessibility assessment, comprehensive of all the possible useful features is still lacking. Investigations of accessibility evaluation tools conducted in the past have actually pointed out various issues [ 2 , 4 , 6 – 8 ]. They already revealed that such tools should provide more support for developers in conducting accessibility audits as well as in understanding and fixing errors.…”