DomeVR: A setup for experimental control of an immersive dome virtual environment created with Unreal Engine 4

Abstract

Immersive virtual reality (VR) environments are a powerful tool to explore cognitive processes ranging from memory and navigation to visual processing and decision making - and to do so in a naturalistic yet controlled setting. As such, they have been employed across different species, and by a diverse range of research groups. Unfortunately, designing and implementing behavioural tasks in such environments often proves complicated. To tackle this challenge, we created DomeVR, an immersive VR environment built using Unreal Engine 4 (UE4). UE4 is a powerful game engine with photo-realistic graphics containing a visual scripting language designed for use by non-programmers. As a result, virtual environments are easily created using drag-and-drop elements. DomeVR aims to make these features accessible to neuroscience experiments. This includes a logging and synchronization system to solve timing uncertainties inherent in UE4; an interactive GUI for scientists to observe subjects during experiments and adjust task parameters on the fly, and a dome projection system for full task immersion in non-human subjects. These key features are modular and can easily be added individually into other UE4 projects. Finally, we present proof-of-principle data highlighting the functionality of DomeVR in three different species: human, macaque and mouse.

Thoughtful faces: inferring internal states across species using facial features

“…1A, top). On the inside of the dome, we projected a virtual reality (VR) environment using a custom-made toolbox called DomeVR (16). The monkeys navigated through the VR environment manually using a trackball; the mice ran on a spherical treadmill, the movements of which were translated into VR movements (for details, see Methods - Experimental Setup ).…”

“…Projecting the VR environment on a dome surrounding the animals enabled both their central and peripheral view to be covered, thereby providing an immersive and realistic VR environment. The VR environment was created using DomeVR, our custom-made toolbox combining photorealistic graphics rendered with Unreal Engine 4, with high timing precision required for neuroscience experiments ((16)).…”

“…We recorded videos of the monkeys’ and mice’ faces during the tasks at 60 Hz using Basler acA640-121gm infrared cameras with a modified version of PylonRecorder2 software (https://gitlab.mpcdf.mpg.de/mpibr/scic/pylonrecorder/PylonRecorder2). Additionally, in the monkeys, eye movements were recorded at 500 Hz using a Grasshopper3 infrared camera and the free eye tracking software iRecHS2 (67) and synchronized with DomeVR (16).…”

“…Virtual reality (VR) environments, for instance, allow researchers to create immersive yet highly controlled experimental settings that can be tailored to different species’ intrinsic sensory capacities and behavioural repertoires (14, 15). For instance, color spectra can easily be adapted to the visibility range of most species; and input devices like treadmills allow animals to interface with the VR in largely intuitive ways that require minimal training (16). As such, VR provides a powerful tool for studying animal behaviour in a highly controlled yet dynamic and ethologically sound context.…”

A hemispheric dome setup for naturalistic visual stimulation in head-fixed mice

“…Our dome setup was inspired by previous work (e.g., Yu and Rosa, 2010;Lopes et al, 2021;Denman et al, 2017Denman et al, , 2018Sibille et al, 2022;Shapcott et al, 2022;Gupta et al, 2023), exploiting hemispheric dome screens to cover large parts of the visual field of the mouse. One challenge in such setups is achieving sufficient light intensity, particularly for UV light, as the screen, mirror, and projector are typically not optimized 29 for these wavelengths.…”