The Structural Geology Query Toolkit for digital 3D models: Design custom immersive virtual field experiences

Needle, Mattathias D.; Mooc, Jacky; Akers, John F.; Crider, Juliet G.

doi:10.1016/j.jsg.2022.104710

Cited by 4 publications

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unity (Unity Technologies), which has been used in VR arenas such as the Antarium (Kócsi et al, 2020), provides a user-friendly environment for creating virtual 3D scenes, and is more robust that Panda3D, with a larger community and better support. Unity is relatively easy to learn, and has been used for creating immersive learning environments (Needle et al, 2022) and for studying peripheral visual field loss in humans, as it provides proper perspective correction (Doyon and Jung, 2023).…”

Section: Introductionmentioning

confidence: 99%

Combining Unity with machine vision to create low latency, flexible, and simple virtual realities

Ogawa,

Aoukar,

Leibbrandt

et al. 2024

Preprint

View full text Add to dashboard Cite

In recent years, virtual reality arenas have become increasingly popular for quantifying visual behaviors. By using the actions of a constrained animal to control the visual scenery, the animal is provided the perception of moving through a simulated environment. As the animal is constrained in space, this allows detailed behavioral quantification. Additionally, as the world is generally computer-generated this allows for mechanistic quantification of visual triggers of behavior.We created a novel virtual arena combining machine vision with the gaming engine Unity. For tethered flight, we enhanced an existing multi-modal virtual reality arena, MultiMoVR (Kaushik et al., 2020) but tracked hoverfly wing movements using DeepLabCut-live (DLC-live, Kane et al., 2020). For trackball experiments, we recorded the motion of a ball that a tethered crab was walking on using FicTrac (Moore et al., 2014). In both cases, real-time tracking was interfaced with Unity to control the movement of the tethered animals’ avatars in the virtual world. We developed a user-friendly Unity Editor interface, CAVE, to simplify experimental design and data storage without the need for coding.We show that both the DLC-live-Unity and the FicTrac-Unity configurations close the feedback loop effectively with small delays, less than 50 ms. Our FicTrac-Unity integration highlighted the importance of closed-loop feedback by reducing behavioral artifacts exhibited by the crabs in open-loop scenarios. We show thatEristalis tenaxhoverflies, using the DLC-live-Unity integration, navigate towards flowers. The effectiveness of our CAVE interface is shown by implementing experimental sequencing control based on avatar proximity to virtual structures.Our results show that combining Unity with machine vision tools such as DLC-live and FicTrac provides an easy and flexible virtual reality (VR) environment that can be readily adjusted to new experiments and species. This can be implemented programmatically in Unity, or by using our new tool CAVE, which allows users to design and implement new experiments without programming in code. We provide resources for replicating experiments and our interface CAVE via GitHub, together with user manuals and instruction videos, for sharing with the wider scientific community.

show abstract

Section: Introductionmentioning

confidence: 99%