Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality

Abstract

The need to improve medical diagnosis and reduce invasive surgery is dependent upon seeing into a living human system. The use of diverse types of medical imaging and endoscopic instruments has provided significant breakthroughs, but not without limiting the surgeon's natural, intuitive and direct 3D perception into the human body. This paper presents a method for the use of Augmented Reality (AR) for the convergence of improved perception of 3D medical imaging data (mimesis) in context to the patient's own anatomy (in-situ) incorporating the physician's intuitive multisensory interaction and integrating direct manipulation with endoscopic instruments. Transparency of the video images recorded by the color cameras of a video see-through, stereoscopic HeadMounted-Display (HMD) is adjusted according to the position and line of sight of the observer, the shape of the patient's skin and the location of the instrument. The modified video image of the real scene is then blended with the previously rendered virtual anatomy. The effectiveness has been demonstrated in a series of experiments at the Chirurgische Klinik in Munich, Germany with cadaver and in-vivo studies. The results can be applied for designing medical AR training and educational applications.

Superman-like X-ray vision: Towards brain-computer interfaces for medical augmented reality

“…It uses a focus and context visualization [2] where augmented objects are shown through a focus window. The setup with the BCI is shown in figure 1.…”

“…While such a visualization improves depth perception, the virtual window hides real objects. In [2] a similar technique was used, where augmented objects do not completely occlude the patient skin, but the visibility of some features is preserved based on surface curvature. For non-medical AR, different solutions to visualize occluded objects have been proposed using e.g.…”

Pictorial AR Tag with Hidden Multi-Level Bar-Code and Its Potential Applications

“…Their surgical team now can easily view the imaging data in real time while they are processing the procedure. In 2007, Christoph Bichlmeier and his team introduced a new AR system for intuitive viewing on the deep-seated anatomy of the patient in real time [23]. They also proposed that the system could be integrated surgical tools for rendering volumetrics of computerised tomography scans or magnetic resonance imaging data directly without wasting time.…”

mirracle: An augmented reality magic mirror system for anatomy education

“…For the organs a segmentation of the VKH is used. The augmentation uses contextual in-situ visualization [2] such that the virtual objects are only shown through a circular window as is shown in figure 1. This leads to a better perception of depth, compared to a simple augmentation of the whole CT.…”