Using virtual reality technology in linguistic research

Abstract

In this paper, we argue that empirical research on genuine linguistic topics, such as on the production of multimodal utterances in the speaker and the interpretation of the multimodal signals in the interlocutor, can greatly benefit from the use of virtual reality technologies.Established methodologies for research on multimodal interactions, like the presentation of pre-recorded 2D videos of interaction partners as stimuli and the recording of interaction partners using multiple 2D video cameras have crucial shortcomings regarding ecological validity and the precision of measurements that can be achieved. In addition, these methodologies enforce restrictions on the researcher. The stimuli, for example, are not very interactive and thus not as close to natural interactions as ultimately desired. Also, the analysis of 2D video recordings requires intensive manual annotations, often frame-by-frame, which negatively affects the feasible number of interactions which can be included in a study.The technologies bundled under the term virtual reality offer exciting possibilities for the linguistic researcher: gestures can be tracked without being restricted to fixed perspectives, annotation can be done on large corpora (semi-)automatically and virtual characters can be used to produce specific linguistic stimuli in a repetitive but interactive fashion. Moreover, immersive 3D visualizations can be used to recreate a simulation of the recorded interactions by fusing the raw data with theoretic models to support an iterative data-driven development of linguistic theories. This paper discusses the potential of virtual reality technologies for linguistic research and provides examples for the application of the methodology.

DeepHandsVR: Hand Interface Using Deep Learning in Immersive Virtual Reality

“…There are various studies currently being conducted on immersive virtual reality to provide users with an interface and experience environment that enable real-world interaction through virtual environments or objects with high immersion, as well as realistic and diverse experiences [1][2][3][4]. Based on these studies, applications are being developed in various fields, such as education, tourism, manufacturing, and entertainment (e.g., gaming) by focusing on the sense of presence that determines how realistic the user feels about the experience of where the user is and what the user is doing.…”

Too Hot to Handle: An Evaluation of the Effect of Thermal Visual Representation on User Grasping Interaction in Virtual Reality

“…Immersive headsets have experienced an unprecedented growth in consumer availability [60], with companies such as Facebook, Microsoft or Samsung enabling accessible VR experiences for the masses. The sector's growth prospects reflect its potential to reshape the way people communicate [64,91], play [43], work [45,50] and learn [38,107], achieving high levels of immersion [10].…”

MINT.tools: tools and adaptors supporting acquisition, annotation and analysis of multimodal corpora

“…Previous experience has shown the combination of VR with linguistic research to be very promising [19]. In particular, VR facilitates data augmentation, which allows automatic extraction of information about the interaction of modelled agents and objects.…”