Socially Distanced: Have user evaluation methods for Immersive Technologies changed during the COVID-19 pandemic?

Abstract

Since the emergence of COVID-19 in late 2019, there has been a significant disturbance in human-to-human interaction that has changed the way we conduct user studies in the field of Human-Computer Interaction (HCI), especially for extended (augmented, mixed, and virtual) reality (XR). To uncover how XR research has adapted throughout the pandemic, this paper presents a review of user study methodology adaptations from a corpus of 951 papers. This corpus of papers covers CORE 2021 A* published conference submissions, from Q2 2020 through to Q1 2021 (IEEE ISMAR, ACM CHI, IEEE VR). The review highlights how methodologies were changed and reported; sparking discussions surrounding how methods should be conveyed and to what extent research should be contextualised, by drawing on external topical factors such as COVID-19, to maximise usefulness and perspective for future studies. We provide a set of initial guidelines based on our findings, posing key considerations for researchers when reporting on user studies during uncertain and unprecedented times.

The reality of remote extended reality research: Practical case studies and taxonomy

“…To this end, it is possible to devise new options for remote XR studies by selecting options from the attributes in the leaf nodes of the taxonomy for each dimension and sub-dimension. We note that although the Finally, we note that this taxonomy describes the most universal attributes which we found relevant in literature (MacKenzie, 2013;Steed et al, 2016Steed et al, , 2020Moran, 2020;Moran and Pernice, 2020;Wiberg et al, 2020;Ratcliffe et al, 2021b;Schmidt et al, 2021;Spittle et al, 2021) and through our implementation of several remote studies detailed above (plus other ongoing and/or unpublished studies). The taxonomy is likely best viewed as a starting point, and we do not claim that it is fully complete.…”