Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Abstract

Current 'dry lab' surgical phantom simulators are a valuable tool for surgeons which allows them to improve their dexterity and skill with surgical instruments. These phantoms mimic the haptic and shape of organs of interest, but lack a realistic visual appearance. In this work, we present an innovative application in which representations learned from real intraoperative endoscopic sequences are transferred to a surgical phantom scenario. The term hyperrealism is introduced in this field, which we regard as a novel subform of surgical augmented reality for approaches that involve real-time object transfigurations. For related tasks in the computer vision community, unpaired cycle-consistent Generative Adversarial Networks (GANs) have shown excellent results on still RGB images. Though, application of this approach to continuous video frames can result in flickering, which turned out to be especially prominent for this application. Therefore, we propose an extension of cycle-consistent GANs, named tempCycleGAN, to improve temporal consistency. The novel method is evaluated on captures of a silicone phantom for training endoscopic reconstructive mitral valve procedures. Synthesized videos show highly realistic results with regard to 1) replacement of the silicone appearance of the phantom valve by intraoperative tissue texture, while 2) explicitly keeping crucial features in the scene, such as instruments, sutures and prostheses. Compared to the original CycleGAN approach, tempCycleGAN efficiently removes flickering between frames. The overall approach is expected to change the future design of surgical training simulators since the generated sequences clearly demonstrate the feasibility to enable a considerably more realistic training experience for minimally-invasive procedures.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

“…Three complementary ways have recently been shown to mitigate this problem. First, if clinically acquired data is available in addition to the well annotated synthetic data, style transfer algorithms can be trained that alter the appearance of real data to close the domain gap, as shown for ophthalmic surgical microscopy [63], [64]. Using such enhanced simulated data for training of more complex tasks has been applied successfully to endoscopy [65] and X-ray imaging [66].…”

Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging

“…In both cases, there is a certain degree of consistency between the successive projections. This can potentially be exploited by simultaneous processing of several projections in order to transfer this consistency to the results [47][48][49] .…”

Cross-Domain Conditional Generative Adversarial Networks for Stereoscopic Hyperrealism in Surgical Training

“…In our previous work [1], a deep learning-based concept to tackle the issue of photo-realism of surgical simulations was presented. The approach was coined hyperrealism, which is able to map patterns learned from intraoperative video sequences onto the video stream captured during simulated surgery on anatomical replica.…”

“…The key to the success of such generative adversarial networks (GANs) is the idea of an adversarial loss that forces the generated images to be, in principle, indistinguishable from real images. Such concepts are able to generate realistic representations of phantoms learned from real intraoperative endoscopic sequences [1]. Conditioned on frames from the surgical training process, the learned models are able to generate impressive results by transforming unrealistic parts of the image (e.g.…”

“…the uniform phantom texture is replaced by the more heterogeneous texture of the tissue). However, the traditional CycleGAN approach [4] neither enforces temporal coherence, which was incorporated in our previous contribution [1], nor enforces a stereo pair to be consistent.…”