Projected Augmented Reality to Display Medical Information Directly on a Patient’s Skin

Abstract

A patient’s internal anatomy can be difficult to visualize when viewed on a monitor, head-mounted display, or even when looking at an actual patient. Combining medical images (CT, MRI, US, PET) with a physical model helps recover missing anatomical context and improves situational awareness. This chapter describes an augmented reality system capable of projecting medical image information directly onto curved targets such as the human body or a mannequin. The motion of the targets and the projector are tracked using a motion capture system so that the images are adjusted in real time to match the anatomy changes in position and orientation. The augmented information can be displayed using volume rendering for realistic visualization of the internal anatomy and 3D models from segmented images. Calibration is performed on the projector and the tracking system to obtain an accurate, common coordinate system and correct visual distortions created by the fact that the projected screen (human body) is no longer a plane. The system is easily extendable to other display technology and has many potential applications, including medical education, surgical planning, and laparoscopic surgery.