New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

Abstract

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

Challenges of intracellular visualization using virtual and augmented reality

“…The combination of this additional information would improve training datasets for deep learning approaches, if, however, the size of the datasets that can be visualized and interacted within VR is increased. D) Finally, if considering VR/AR approaches as an interesting way to improve microscopy data annotation, feedback information must be linked to the input dataset, meaning refreshing a database connected to the VR system, via Cloud technology ( Guérinot et al, 2022 ). This could be potentially achieved close to real time, through a hybrid technology, combining VR/AR and visualization on 2D screens.…”

“…D) Finally, if considering VR/AR approaches as an interesting way to improve microscopy data annotation, feedback information must be linked to the input dataset, meaning refreshing a database connected to the VR system, via Cloud technology ( Guérinot et al, 2022 ). This could be potentially achieved close to real time, through a hybrid technology, combining VR/AR and visualization on 2D screens.…”

“…(C) Dynamic exchange of mitochondria between cells using Tunneling Nanotubes (TNT) (Courtesy of Chiara Zurzolo’s lab, Institut Pasteur) visualized with DIVA in VR (see Supplementary Video S1 ). Rendering is performed by full volumetric ray casting (el Beheiry et al, 2020 ) and an adapted transfer function ( Guérinot et al, 2022 ). Note the possibility to have both the video running and the user interacting with the data.…”

Metaverse Wearables for Immersive Digital Healthcare: A Review

“…Copyright 2021, BioMed Central (BMC). Intracellular imaging: Reproduced with permission [128]. Copyright 2022, Frontiers.…”

“…g) A rendering image of mouse hippocampus captured by two-photon microscopy. Reproduced with permission [128]. Copyright 2022, Frontiers.…”