Remote Evaluation in Universal Design Using Video Conferencing Systems During the COVID-19 Pandemic

Abstract

Usability and accessibility evaluations with diverse users are an essential part of an iterative universal design process for digital solutions. The COVID-19 pandemic has made it difficult to run traditional local evaluations due to social distancing restrictions to reduce infections. Remote synchronous and asynchronous evaluation methods may be a solution if it can be used by various user groups, including people with impairments. Incorporating video conferencing systems into traditional remote evaluations can be a valuable supplement to -or, given the current situation, alternative to -traditional usability and accessibility evaluations. We present a protocol for remote formative usability evaluations designed with accessibility in mind. The protocol explains how to prepare for an evaluation and its technical setup, how to conduct synchronous evaluations using video conferencing systems, and how to debrief and analyze the collected data. We tested this protocol in a pilot study for an ongoing project where we ran both synchronous and asynchronous remote evaluations. In our pilot, the synchronous evaluation using video conferencing systems provided opportunities for richer qualitative data than the asynchronous evaluation as we ran it. The findings from the pilot study indicate that the protocol is feasible and can be used when having participants with diverse abilities or impairments. We also provide suggestions for others that wish to adapt the protocol.

From the Lab to People's Home: Lessons from Accessing Blind Participants' Interactions via Smart Glasses in Remote Studies

“…After the outbreak of COVID-19 [37], many accessibility researchers switched to a remote format for their user studies and shared best practices with the community [14,45,53]. With a focus on remote studies that involve people with visual impairments over the past two years (Table 1), we observe that common approaches employed by researchers spanned online surveys, audio calls, and • laptop: audio call video conferencing.…”

“…Cameras available on laptops or attached to desktops with video conferencing tools, such as Google Meet [21], Microsoft Teams [35], and Zoom [56], have enabled experimenters to observe participants' interactions [25], which can be supplementary to traditional remote methods for usability testing [45]. Similar setups have also been employed in remote usability testing with blind participants [49].…”

Involving Diverse Users for Inclusive Technology Development

“…Recent work has investigated the use of video conferencing systems in remote evaluations of usability and accessibility of web applications (Simon-Liedtke et al, 2021). Because of the Covid-19 pandemic, many people with and without impairments who had previously not used video conferences, have gained experience with these collaboration tools.…”

“…The researchers found no clear favorite method or system, which highlights the need to be flexible in order to accommodate diverse users. They conclude that conducting remote evaluations of the usability and accessibility of web applications can be feasible for participants with different impairments including people using assistive devices such as screen readers, provided that the tools and materials used are accessible and known to the users in question (Simon-Liedtke et al, 2021).…”