SITbench 1.0: A Novel Switch-Based Interaction Technique Benchmark

Abstract

Evaluation process of a switch-based interaction technique (SIT) requires an interdisciplinary team effort and takes a considerable amount of time. Collecting subjective evaluation data from the users is a very common approach, but the subjective evaluation data alone might be manipulated and unreliable for comparing performances in many cases. Thus, therapists generally cannot succeed in determining the optimum SIT setup (i.e., determining the most appropriate combination of setup variables such as the switch type or switch site) at first attempts since it is hard to evaluate the measurable performance by collecting subjective data instead of objective data. Inevitably, each unsuccessful attempt to reach the optimum SIT setup results in a loss of serious time and effort. On the contrary, a benchmark application is also required to make performance evaluation of SITs by using a number of standard tests and empirical attributes. It is obvious that a quicker and more accurate SIT evaluation process provides a better cost and schedule management considering the increasing number of SIT users in the world. Therefore, we propose a novel benchmark for performance evaluation called SITbench that provides a quicker and more accurate switch evaluation process by collecting and saving the objective data automatically. We conducted a user study with eight participants and demonstrated that the objective data collected via the SITbench helped to determine the optimum SIT setup accurately. Result of a questionnaire applied to evaluate the SITbench itself was also satisfactory. SITbench is expected to help researchers and therapists to make a better evaluation according to any change done in SIT setup variables (switch type, activation method, etc.) with the aim of reaching the optimum SIT setup, which leads to a better cost and schedule management. As the first benchmark application compatible with all SITs, which can emulate keyboard characters or mouse clicks, it can be utilized by assistive technology professionals to make comparisons and evaluations automatically via standardized tests.

Novel hands-free interaction techniques based on the software switch approach for computer access with head movements

“…We used the SITbench 1.0 [7] benchmark which helps researchers to evaluate switch-based systems objectively. By means of this tool, objective evaluation data can be collected and saved automatically with standardized tests.…”

“…A usability study with 36 participants (18 motorimpaired, 18 able-bodied) was conducted in order to collect objective and subjective evaluation data. The SITbench 1.0 [7] benchmark was employed for objective evaluation. Moreover, we also applied a System Usability Scale (SUS) [8] questionnaire for subjective evaluation.…”

Erratum to “SITbench 1.0: A Novel Switch-Based Interaction Technique Benchmark”

“…In the article titled “SITbench 1.0: A Novel Switch-Based Interaction Technique Benchmark” [1], there was a structural error in the fifth sentence of the Abstract, where the phrase “On the contrary” should be written as “On the other hand.” Therefore, the fifth sentence should be updated as follows:…”