Short Scales of Satisfaction Assessment: A Proxy to Involve Disabled Users in the Usability Testing of Websites

Abstract

Abstract. Short scales of user satisfaction analysis are largely applied in usability studies as part of the measures to assess the interaction experience of users. Among the traditional tools, System Usability Scale (SUS), composed of 10 items, is the most applied quick evaluation scale. Recently, researchers have proposed two new and shorter scales: the Usability Metric for User Experience (UMUX), composed of four items, and the UMUX-LITE, which consists of only the two positive items of UMUX. Despite their recent creation, researchers in human-computer interaction (HCI) have already showed that these two tools are reliable and strongly correlated to each other [1][2][3]. Nevertheless, there are still no studies about the use of these questionnaires with disabled users. As HCI experts claim [4][5][6][7], when disabled and elderly users are included in the assessment cohorts, they add to the overall analysis alternative and extended perspectives about the usability of a system. This is particularly relevant to those interfaces that are designed to serve a large population of end-users, such as websites of public administration or public services. Hence, for a practitioner adding to the evaluation cohorts a group of disabled people may sensibly extend number and types of errors identified during the assessment. One of the major obstacles in creating mixed cohorts is due to the increase in time and costs of the evaluation. Often, the budget does not support the inclusion of disabled users in the test. In order to overcome these hindrances, the administering to disabled users of a short questionnaire-after a period of use (expert disabled costumers) or after an interaction test performed through a set of scenario-driven tasks (novice disabled users)-permits to achieve a good trade-off between a limited effort in terms of time and costs and the advantage of evaluating the user satisfaction of disabled people in the use of websites. To date, researchers have neither analyzed the use of SUS, UMUX, and UMUX-LITE by disabled users, nor the reliability of these tools, or the relationship among those scales when administrated to disabled people. In this paper, we performed a usability test with 10 blind and 10 sighted users on the Italian website of public train transportation to observe the differences between the two evaluation cohorts in terms of: (i) number of identified errors, (ii) average score of the three questionnaires, and (iii) reliability and correlation of the three scales.The outcomes confirmed that the three scales, when administered to blind or sighted users, are reliable (Cronbach's α > 0.8), though UMUX reliability with disabled users is lower than expected (Cronbach's α < 0.5). Moreover, all the scales are strongly correlated (p < .001) in line with previous studies. Nevertheless, significant differences were identified between sighed and blind participants in terms of (i) number of errors experienced during the interaction and (ii) average satisfaction rated through the three questionnaires. Our data ...

Preliminary Results of a Systematic Review: Quality Assessment of Conversational Agents (Chatbots) for People with Disabilities or Special Needs

“…(i) Researchers tend to consider a lack of complaints as an indirect measure of the safety and acceptability of tools. However, safety and acceptability should be assessed with consolidated and comparable methodologies to rule out risks in use [ 37 – 39 ]. (ii) Satisfaction, intended as a usability metric, is a different construct from acceptability, and these two constructs should be measured separately with available standardized questionnaires [ 39 , 40 ].…”

“…However, safety and acceptability should be assessed with consolidated and comparable methodologies to rule out risks in use [ 37 – 39 ]. (ii) Satisfaction, intended as a usability metric, is a different construct from acceptability, and these two constructs should be measured separately with available standardized questionnaires [ 39 , 40 ]. (iii) Although dedicated tools and methods for assessing the quality of interaction with chatbots are lacking, reliable methods and measures to assess interaction are available [ 17 , 19 , 21 , 37 ], and these should be adopted and used to enable the generation of comparable evidence regarding the quality of conversational agents.…”