On the Multimodal Resolution of a Search Sequence in Virtual Reality

Abstract

In virtual reality (VR), participants may not always have hands, bodies, eyes, or even voices—using VR helmets and two controllers, participants control an avatar through virtual worlds that do not necessarily obey familiar laws of physics; moreover, the avatar’s bodily characteristics may not neatly match our bodies in the physical world. Despite these limitations and specificities, humans get things done through collaboration and the creative use of the environment. While multiuser interactive VR is attracting greater numbers of participants, there are currently few attempts to analyze the in situ interaction systematically. This paper proposes a video-analytic detail-oriented methodological framework for studying virtual reality interaction. Using multimodal conversation analysis, the paper investigates a nonverbal, embodied, two-person interaction: two players in a survival game strive to gesturally resolve a misunderstanding regarding an in-game mechanic—however, both of their microphones are turned off for the duration of play. The players’ inability to resort to complex language to resolve this issue results in a dense sequence of back-and-forth activity involving gestures, object manipulation, gaze, and body work. Most crucially, timing and modified repetitions of previously produced actions turn out to be the key to overcome both technical and communicative challenges. The paper analyzes these action sequences, demonstrates how they generate intended outcomes, and proposes a vocabulary to speak about these types of interaction more generally. The findings demonstrate the viability of multimodal analysis of VR interaction, shed light on unique challenges of analyzing interaction in virtual reality, and generate broader methodological insights about the study of nonverbal action.

Reducing Stigma Toward People with Serious Mental Illness Through a Virtual Reality Intervention: A Randomized Controlled Trial

“…Third, in our study only one stigma measurement instrument was used, and because stigma is a complex construct, future research should assess the effect on other cognitive, emotional, and behavioral components of the construct (e.g., by measuring implicit stigma). Fourth, our study did not include within the design and analysis mediating variables of the mechanisms of interaction, perception, and behavior in the virtual reality environment that can affect and explain the effects of the intervention, such as the degree of immersion and sense of presence 44 nor the sequence and time of interaction with the virtual environments 45 , so future studies should include these variables. Finally, we did not account for the specific characteristics of the participants.…”