Experiences with an AR evaluation test bed: Presence, performance, and physiological measurement

Abstract

This paper discusses an experiment carried out in an AR test bed called "the pit". Inspired by the well-known VR acrophobia study of Meehan et al. [18], the experimental goals were to explore whether VR presence instruments were useful in AR (and to modify them where appropriate), to compare additional measures to these well-researched techniques, and to determine if findings from VR evaluations can be transferred to AR. An experimental protocol appropriate for AR was developed. The initial experimental findings concern varying immersion factors (frame rate) and their effect on feelings of presence, user performance and behavior. Unlike the VR study, which found differing frame rates to affect presence measures, there were few differences in the five frame rate modes in our study as measured by the qualitative and quantitative instruments, which included physiological responses, a custom presence questionnaire, task performance, and user behavior. The AR presence questionnaire indicated users experienced a high feeling of presence in all frame rate modes. Behavior, performance, and interview results indicated the participants felt anxiety in the pit environment. However, the physiological data did not reflect this anxiety due to factors of user experience and experiment design. Efforts to develop a useful AR test bed and to identify results from a large data set has produced a body of knowledge related to AR evaluation that can inform others seeking to create AR experiments. INTRODUCTIONFor many years the technological challenges of AR were all consuming. Often the biggest hurdle was creating a working application using complicated tools and bulky homemade hardware. Over time these obstacles were overcome and researchers began to look more at applications and interfaces.Researchers began to bring techniques from HCI into the field and started performing formal evaluations and experiments on AR systems to learn more about effective interfaces, compelling presentation strategies, collaboration, and how to craft engaging user experiences [7] [19]. However, AR is a relatively young field;there is still extensive work needed in both the development of evaluation techniques that are appropriate for AR and in building an understanding of how various factors of an AR experience from display techniques to interfaces impact a user's effectiveness, immersion, productivity, and enjoyment. With these goals in mind we have spent the last four years building and experimenting in an AR test bed we call "the pit". This test bed has provided a platform for exploring approaches to the rendering and presentation of augmentations as well as a controlled setting for performing experiments. By creating an augmented experience that was designed to cause physical reactions in the user, we hoped to utilize physiological and behavioral measurements in the experiment to learn how anxiety can modify a users performance and feelings of presence. The motivations of this project revolve around our desire to build an understanding of AR evaluati...

Online user survey on current mobile augmented reality applications

“…A recent paper by Gandy et al [12] presents an interesting multi-method setup for inspecting how immersion factors affect the aspects of presence, performance, and physiological responses of a user. They gathered physiological data, task performance data, and subjective questionnaire data with a case-specific questionnaire and NASA TLX, as well as interview data.…”

“…As emphasised in Gandy et al [12], AR as a field of research is still lacking evaluation methods that are appropriate for the specific technology in question. We agree that there is a challenge involved in utilising metrics that are too abstract and on too general a level in evaluating specific technologies.…”

“…The content aspect is supported by 14 responses, implying that the applications are not yet common enough, meaning e.g. a lack of user-generated content (12). Four responses mentioned that the application with such content has not reached its full potential.…”

Burnar: Involuntary heat sensations in augmented reality

“…As reported in previous work on evaluating AR systems [9], the enthusiastic responses from casual demonstrations may not be replicable in formal experiments. This necessitates that the evaluation of the responses reported from the informal BurnAR public demonstrations need to be investigated in a tightly controlled environment, minimizing the effects of autosuggestion and prior knowledge.…”

Presence and discernability in conventional and non-photorealistic immersive augmented reality

“…Our second experimental task, described in Section 4.2 is inspired by acrophobia experiments studied in VR [13], and later trialled in immersive AR [6]. In those studies, participants report feeling frightened and some report vertigo when faced with the virtual pit.…”