Users' performance in lab and non-lab enviornments through online usability testing: A case of evaluating the usability of digital academic libraries' websites

Abstract

The factors related to the environment in which users operate may be of a vital importance when trying to understand how they experience a particular system. It is required that we find out how we can get to know those factors to investigate if they affect the users' performance in usability testing. An online usability study has emerged that can be attempted by a large, varied pool of users' anywhere with an Internet connection. Would the usage of an online usability study help to give comprehensive insight and an understanding of the whole user experience? That is especially interesting if the user operates remotely, as we are unaware of what the users might experience while performing the test (e.g., distractions and type of device used to attempt the test). Accordingly, a pilot study of ongoing research was conducted. An identical online usabilitytesting tool (Loop11) through which to apply the online usability study was used in two environments: unrestricted (the user's natural environment), and totally restricted (a simulated lab environment). Ten subjects completed the test in the restricted environment and 20 completed it in the unrestricted environment. All of the subjects were asked to perform predefined search tasks on digital libraries' websites. Their performance was analyzed and compared against the two different environments. The results showed that online usability testing is a feasible method to gain comprehensive insight into how users attempt usability testing in a non-lab environment. The results of whether different environments affect test performance show no valuable differences in most of the study's measurements. The test subjects were frequently multitask while they performed the usability testing in an unrestricted environment, but they were highly distracted if they personally interrupted. The results encourage the researcher to conduct a formal version of this study to further examine the learned lessons from the pilot study.

Authentic versus synthetic: An investigation of the influences of study settings and task configurations on search behaviors

“…For example, Kelly and Gyllstrom (2011) did not find differences between lab and remote participants for the behavioral measures, though great variances were observed for task completion times and the number of documents opened and saved in the remote group. Alharbi and Mayhew (2015) and Greifeneder (2016) also reported no difference in task completion time between lab and remote participants, despite the distractions faced by participants in the field.…”

Novel ethnographic/contextual inquiry techniques for understanding connected device users in their native environment

“…People regularly broadcast what they are eating, who they are with, and what they are doing, all in context of their daily lives. Consumer product companies have taken advantage of this trend, and have begun to use remote usability techniques to study mobile application users in their natural context (Alharbi & Mayhew, 2015;Bastien, 2010;Burzacca & Paternò, 2013;Chalil Madathil & Greenstein, 2011;De Guzman, 2016;Rodriguez & Resnick, 2010). However, remote research techniques have not yet been employed in a systematic manner to study physical medical devices or their associated MMAs in a regulated environment.…”