Argumentation Models for Usability Problem Analysis in Individual and Collaborative Settings

Abstract

Consolidating usability problems from problem lists from several users can be a cognitively demanding task for evaluators. It has been suggested that collaboration between evaluators can help this process. In an attempt to learn how evaluators make decisions in this process, we studied what justification evaluators give for extracting usability problems and their consolidation when working both individually and collaboratively. An experiment with eight novice usability evaluators was carried out where they extracted usability problems and consolidated them individually and then collaboratively. The data were analysed by using conventional content analysis and by creating argumentation models according to the Toulmin model. The results showed that during usability problem extraction novice usability evaluators could put forward warrants leading to clear claims when probed, but seldom added qualifiers or rebuttals. Novice usability evaluators could identify predefined criteria for a usability problem when probed and this could be acknowledged as a backing to warrants. In the individual settings, novice evaluators had difficulty in presenting claims and warrants for their decisions on consolidation. Although further study is needed, the results of the study indicated that collaborating pairs had a tendency to argue slightly better than individuals. Through the experiment novice evaluators' reasoning patterns during problem extraction and consolidation as well as during their assessment of severity and confidence could be identified.

Consolidation in Collaborative Design: An Exploratory Case Study

“…In usability evaluation, Hvannberg et al (2019) divide Consolidation into filtering duplicate usability issues from one list and merging usability issues that could be retained, merged or discarded. For Pohl and Sikora (2007), Consolidation is a sub-process of requirements co-design and software architecture composed by: (1) categorization, (2) prioritization, (3) correction of inconsistencies and (4) a stage of comparison and refinement where visions are integrated and new ideas are generated.…”

“…Consolidation is a process of combining, integrating or transforming something into something else that is more complete, effective, coherent or elaborated. Consolidation is a common activity in software design and implementation processes and can be applied in several contexts, such as consolidating an idea [Aiken and Carlisle 1992] or usability problems [Hvannberg et al 2019].…”