Mobile guide technologies (smartphone apps): Collaborative Heuristic Evaluation (CHE) with expert and novice users

Abstract

Providing quality and meaningful experiences to the users are vital particularly with the devices that users are familiar with. Usability issues should be discarded before the systems/designs are deployed to the users. In this study, we compared the usability issues found by expert and novice users using Collaborative Heuristic Evaluation (CHE). This study was carried out with 6 experts and 9 novice users. There are 3 different mobile guides used in this study, 2 were designed to be used in museum and another mobile guide was designed to be used at a tourist attraction. Results showed that expert users found significantly more heuristic problems than novice users during the CHE procedure across the 3 different mobile guides (smartphone apps). On the other hand, there were heuristic problems found by the novices that were not noticeable to the expert.

Adaptations to the Heuristic Evaluation (HE) method for novice evaluators

“…The union of reports G3 and G4 has ten (10) hits out of the 25 usability problems listed in G7 (40% of the usability problems listed in report G7). This coverage of novice reports in expert reports is higher than the coverage described by Othman et al (2014), when novice reports listed from 30% to 35% of the problems listed in expert reports. Also, the union of reports G3 and G4 had 15 false alarms.…”

“…As shown in the figure, G3 had four (4) hits while G4 had no one. In this case, the union of reports from level 1 cover less than 11% of problems listed in report G7, which is about one/third of the coverage showed by Othman et al (2014) for groups composed only by novice evaluators. On the other side, the matching criteria adopted by Othman et al 2014is not clear enough.…”

“…On the other hand, G1 and G2 had one (1) distinct hit each. The union of both reports (G1 and G2) resulted in two (2) hits (≈ 5% of the problems listed in G7), which is much less than the coverage showed by Othman et al (2014). In addition, such union had 11 false alarms (≈ 85% of problems listed in both G1 and G2).…”

“…Also, according to Huang (2012), novices were not always precise in describing problems. Othman et al (2014) investigated the difference between the performance of a group of novice evaluators and a group of expert evaluators conducting a CHE of 3 mobile applications. In contrast to Huang (2012), Othman et al (2014) did not compare the outcomes from CHEs with outcomes from test with users.…”

“…Othman et al (2014) organized five (5) groups of evaluators, but their composition (in terms of evaluator expertise) was not clear. In the case study of Othman et al (2014), novice evaluators covered between 30% to 35% of the usability problems reported by expert evaluators. Although Huang (2012) and Othman et al (2014) investigated aspects of the participation of novice evaluators in CHE, exploring the performance of mixed groups (groups composed by both expert and novice evaluators) remains as a gap in the field.…”