Rethinking self-reported measure in subjective evaluation of assistive technology

Abstract

IntroductionSubjective evaluation of assistive technology solutions often has access to limited number of representative users in the rating process. Sometimes non-representative users (e.g. a blind folded person as blind) act as subject to increase the number. Most of the case same data collection process, self-reported measures (e.g. questioners) are used for both [1] groups of users. In the data collection process, representative users need to express their subjective opinion to data collector (facilitator). This can introduce cognitive bias and may hinder the relatively unbiased data collection. The intermediary assistance causes direct or indirect influence to the rating process and makes the rating process slower and less objective. It also affects the "privacy" and may have issues, such as 'Forer effect' [2] or 'Hawthorne effect' [3]. Figure 1 shows a schematic diagram of subjective data collection. A representative subject is participating in the rating process with the help of a 'facilitator' . In step one, the facilitator may ask a question to the subject with subject's preferred medium of instruction (audio, Braille etc.). The subject may understand the question (step 2) and answer with a subjective score (step 3) or may not able to answer due to medium or other problem. If subject answer the question, the next step of the facilitator is to record the score on behalf of subject (step 4) or repeat the question (if necessary changing the medium). AbstractSelf-reporting is used as a subjective measure of usability study of technology solutions. In assistive technology research, more than often the 'a coordinator' directly assist the 'subject' in the scoring process. This makes the rating process slower and also introduces bias, such as, 'Forer effect' and/or 'Hawthorne' effect. To address these issues we propose to use technology mediated interaction between the 'subject' and 'the coordinator' in evaluating assistive technology solutions. The goal is to combine both the qualitative and quantitative scores to create a relatively unbiased rating system. Empirical studies were performed on two different datasets in order to illustrate the utility of the proposed approach. It was observed that, the proposed hybrid rating is relatively unbiased for usability study.

User’s Demographic Characteristic on the Evaluation of Gamification Interactive Typing for Primary School Visually Impaired with System Usability Scale

“…Cronbach's Alpha [21] calculation is conducted to examine the reliability and validity of SUS. Hossain reported that reliability is Female 46% Male 54% acceptable when 0.6 ≤ α ≤ 0.7 [25]. This finding indicates that this SUS is reliable (α = 0.601).…”

Analysis of MOOC Quality Requirements for Landscape Architecture Based on the KANO Model in the Context of the COVID-19 Epidemic

“…Generally, if the Cronbach's alpha is above 0.9, the reliability of the questionnaire is excellent; if it is above 0.7, the reliability is good; if it is above 0.6, the reliability is acceptable; if it is below 0.6, the scale shall be redesigned. In the reliability test of the questionnaire, the Cronbach's alpha of the forward and reverse questions exceeds 0.9 on the whole, and the Cronbach's alpha of the forward and reverse questions in most functional modules is above 0.6, which is acceptable [49] (Table 4).…”

Key Factors Influencing Chinese Consumers’ Demand for Naturally Dyed Garments: Data Analysis through KJ Method and KANO Model

“…Reliability was represented by Cronbach's alpha coefficient, ranging from 0 to 1, with higher values indicating greater internal consistency of the questionnaire. Many studies mention that a coefficient of 0.7 or above indicates effective reliability, with some considering 0.6 or above acceptable [74]. As per Table 8, the overall Cronbach's alpha coefficient for the 387 valid questionnaires was 0.855, with 0.964 for positive questions and 0.969 for negative questions, all above 0.8.…”