Evidence Based Design of Heuristics for Computer Assisted Assessment

Abstract

Abstract. The use of heuristics for the evaluation of interfaces is a well studied area. Currently there appear to be two main research areas in relation to heuristics: the analysis of methods to improve the effectiveness of heuristic evaluations; and the development of new heuristic sets for novel and specialised domains. This paper proposes an evidence based design approach to the development of domain specific heuristics and shows how this method was applied within the context of computer assisted assessment. A corpus of usability problems was created through a series of student surveys, heuristic evaluations, and a review of the literature. This corpus was then used to synthesise a set of domain specific heuristics for evaluating CAA applications. The paper describes the process, and presents a new set of heuristics for evaluating CAA applications.

Using computer‐assisted assessment heuristics for usability evaluations

“…The aggregated forms were analysed by the first author and an educational technologist to verify that the problems predicted were not false positives. The heuristic evaluation method is an inspection method where evaluators are required to predict problems that they think users will encounter; this means that on occasions, usability problems can be included in the aggregated list that are not real problems in that a user would not have a problem, hence the phrase “false positives.” To understand the effectiveness of the evaluators, the aggregated lists were compared with an earlier corpus of known usability problems that were derived to synthesise the original heuristic set (Sim et al , ). This corpus consisted of 34 usability problems synthesised from an initial corpus of over 300 usability problems that were systematically merged and filtered to produce the final corpus.…”

“…Nielsen's heuristics were used for the evaluation of Questionmark Perception and WebCT (Sim, Read & Holifield, ), and despite the fact that usability problems were revealed, the heuristic set was found to be ineffective, as problems were not mapped to heuristics or given a severity ratings. Therefore, using a qualitative research approach, a set of heuristics was synthesised based upon a corpus of usability problems associated with CAA (Sim, Read & Cockton, ). Thematic analysis was then used to identify the core themes from the corpus, and these were then used to create the heuristic set presented in Table .…”

The Design of Application-Specific Heuristics for the Usability Evaluation of the Digital Doorway

“…Indeed, Nielsen (as cited in Rogers et al [44]), acknowledges the inapplicability of this set to contemporary applications, and suggests developing application-specific heuristics when necessary. A wide number of application-specific or domain heuristics have emerged, includiFng the Pierotti's Xerox Heuristic Evaluation Checklist [41]; Travis's web usability guidelines [53]; Desurvire et al's heuristics for evaluating the playability of games [14]; and, Sim et al's heuristics for evaluating the usability of computer-assisted assessment (CAA) applications [48]. One of the most prominent sets of heuristics, prior to Nielsen coining the term 'heuristic evaluation', is most probably the 1986 set of guidelines for designing interface software by Smith and Mosier developed for the United States Air Force Systems Command [50].…”

“…Furthermore, Sim et al [48] defined criteria that can be used to assess the effectiveness of application-specific heuristics, in addition to the number of usability problems identified through the use of such heuristics. These criteria are correctness of terminology, and coverage and thoroughness.…”

“…As a final step in determining the effectiveness of the application of the multi-category heuristics, we assessed them against Sim et al [48] criteria of correctness of terminology, and coverage and thoroughness:…”