Interactive Visualization and Analysis of Multimodal Datasets for Surgical Applications

Abstract

Surgeons use information from multiple sources when making surgical decisions. These include volumetric datasets (such as CT, PET, MRI, and their variants), 2D datasets (such as endoscopic videos), and vector-valued datasets (such as computer simulations). Presenting all the information to the user in an effective manner is a challenging problem. In this paper, we present a visualization approach that displays the information from various sources in a single coherent view. The system allows the user to explore and manipulate volumetric datasets, display analysis of dataset values in local regions, combine 2D and 3D imaging modalities and display results of vector-based computer simulations. Several interaction methods are discussed: in addition to traditional interfaces including mouse and trackers, gesture-based natural interaction methods are shown to control these visualizations with real-time performance. An example of a medical application (medialization laryngoplasty) is presented to demonstrate how the combination of different modalities can be used in a surgical setting with our approach.

Surgical planning of Isshiki type I thyroplasty using an open-source Digital Imaging and Communication in Medicine viewer OsiriX

“…Multimodal visualization techniques have been used to present all available information in an effective manner [10,11]. A series of studies related to image-guided medialization laryngoplasty were published by Kirmizibayrak et al [3], Jin et al [12], and Yim et al [13]. They established new programs specific to laryngoplasty that would be useful for such a procedure.…”

“…One of the reasons for this high failure rate is that vocal folds are not visible to the surgeon during the procedure, leading to the need for educated guesses based on medical imaging (i.e. CT scans of the patient and laryngoscope videos) and the surgeon's domain knowledge [3]. Prof. Isshiki reported that the anterior commissure exists at the midpoint between the thyroid notch and the lower margin of the thyroid cartilage.…”

Combined Volume Registration and Visualization

“…In [KYWH12], the authors present a system they call "magic lens visualization," providing overall context and specialized rendering at the focus of attention. A similar system is described in [DPLR10].…”

Generalized Temporal Focus + Context Framework for Improved Medical Data Exploration

“…The first application of a magic lens in the medical domain for an improved dataset exploration was proposed by Viega et al [2]. Other researchers have further extended the focus+context paradigm to multiple volumetric lenses [16], various volume distortions-e.g., zooming [17][18][19], and different techniques for volume rendering [20][21][22].…”