Towards Mobile Augmented Reality for On-Patient Visualization of Medical Images

Abstract

Abstract. Despite considerable technical and algorithmic developments related to the fields of medical image acquisition and processing in the past decade, the devices used for visualization of medical images have undergone rather minor changes. As anatomical information is typically shown on monitors provided by a radiological work station, the physician has to mentally transfer internal structures shown on the screen to the patient. In this work, we present a new approach to on-patient visualization of 3D medical images, which combines the concept of augmented reality (AR) with an intuitive interaction scheme. The method requires mounting a Time-of-Flight (ToF) camera to a portable display (e.g., a tablet PC). During the visualization process, the pose of the camera and thus the viewing direction of the user is continuously determined with a surface matching algorithm. By moving the device along the body of the patient, the physician gets the impression of being able to look directly into the human body. The concept can be used for intervention planning, anatomy teaching and various other applications that require intuitive visualization of 3D data.

Mobile markerless augmented reality and its application in forensic medicine

“…As proposed in [29], we provide another shader to cut a hole in the camera center to obtain an "X-ray vision" like effect. The hole size can be adapted.…”

“…In previous publications [22,28,29], we introduced the general concept of mobile AR using range imaging but had to complete the performance assessment with simulated data due to the lack of appropriate hardware (portable device) and software (real-time registration). In this work, we (1) propose a practical (real-time) implementation of the concept using a general-purpose computing on graphics processing units (GPGPU) [24] approach and (2) apply it for intuitive visualization of multimodal forensic data.…”

Focus plus context visualization based on volume clipping for markerless on-patient medical data visualization

“…Some of them do not run in real-time (more than 15 frames per second, FPS) [9,10] and others rely on specific prior knowledge about the ROI to be tracked (see [11][12][13] for the body and [14] for the face). To the best of our knowledge, there is only one exception which can be used for general-purpose markerless on-patient medical data visualization: the semiautomatic approach proposed in [15][16][17].…”

“…Considering that the bone is rendered with a gray level greater than the soft tissue's and than w grayLevel , it is rendered without the background scene. Assuming that bone and soft tissue have different gray intensities, w grayLevel can be adjusted to render the bone with its original color and the soft tissue can be linearly interpolated with the background scene (lines [8][9][10][11][12][13][14][15].…”

Real-Time Range Imaging in Health Care: A Survey

“…In particular, low-cost RI cameras hold potential for cost-sensitive applications such as medical education, training and rehabilitation. One promising application involves tracking the pose of a person in order to visualize subsurface anatomical detail via AR, as suggested by Maier-Hein et al [83] and Blum et al [84]. Maier-Hein et al [83] proposed mounting a ToF camera to a portable display or tablet for on-patient visualization of medical images, as shown in Fig.…”