Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

Rivoir, Dominik; Pfeiffer, Mark; Docea, Reuben; Kolbinger, Fiona R.; Riediger, Carina; Weitz, Jürgen; Speidel, Stefanie

doi:10.1109/iccv48922.2021.00333

Cited by 19 publications

(11 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our future work includes incorporating virtual fixture testing in the simulation environment (Li et al (2020b), Li et al (2020a)) and a formal user study with surgeons and residents to evaluate the efficacy of our simulator in surgical training. We also recognize the limitation of visual realism and plan to explore learning-based style transfer (e.g., Rivoir et al (2021)) to mitigate the problem.…”

Section: Discussionmentioning

confidence: 99%

Virtual Reality for Synergistic Surgical Training and Data Generation

Munawar,

Li,

Kunjam

et al. 2021

Preprint

View full text Add to dashboard Cite

Surgical simulators not only allow planning and training of complex procedures, but also offer the ability to generate structured data for algorithm development, which may be applied in image-guided computer assisted interventions. While there have been efforts on either developing training platforms for surgeons or data generation engines, these two features, to our knowledge, have not been offered together. We present our developments of a cost-effective and synergistic framework, named Asynchronous Multibody Framework Plus (AMBF+), which generates data for downstream algorithm development simultaneously with users practicing their surgical skills. AMBF+ offers stereoscopic display on a virtual reality (VR) device and haptic feedback for immersive surgical simulation. It can also generate diverse data such as object poses and segmentation maps. AMBF+ is designed with a flexible plugin setup which allows for unobtrusive extension for simulation of different surgical procedures. We show one use case of AMBF+ as a virtual drilling simulator for lateral skull-base surgery, where users can actively modify the patient anatomy using a virtual surgical drill. We further demonstrate how the data generated can be used for validating and training downstream computer vision algorithms. *

show abstract

Section: Discussionmentioning

confidence: 99%

Virtual Reality for Synergistic Surgical Training and Data Generation

Munawar,

Li,

Kunjam

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In SDS, such high-volume information stream has to be acquired and stored, which involves several challenges, e.g., regarding interoperability or standards for storage [28]. Based on big data methods, new ML and AI applications can be developed, where possible deployment domains range from semiautomation of surgical tasks to context-aware surgical guidance [50]- [54].…”

Section: Sds-data and Ai As Enabling Components For Democratizing Ramismentioning

confidence: 99%

Robot-Assisted Minimally Invasive Surgery—Surgical Robotics in the Data Age

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…The use of simulation environments to generate input images has been reported in colonoscopy [26], [27] and in MIS [28], [29], where cGANs, cycle-GANs and their byproducts, such as MUNIT [30], are used to translate realistic features on labelled simulation samples to render synthetic datasets. Differently, [13] and [9] apply I2I on simulated instruments blended onto real surgical backgrounds to produce training frames for tool segmentation.…”

Section: Literature Review: I2i In Surgerymentioning

confidence: 99%

“…In the end, moving away from surgical instrument segmentation, we believe the proposed approach can fit all I2I settings that involve transferring features onto simulation images, where labels of different nature (segmentation, pose, depth) are provided automatically. Also, there should be a clear difference between foreground and background in order for the proposed losses to work appropriately: possible applications may include organ segmentation [29] or robot/human pose estimation [50], [51].…”

Section: E Future Directionsmentioning

confidence: 99%

SSIS-Seg: Simulation-Supervised Image Synthesis for Surgical Instrument Segmentation

Colleoni¹,

Psychogyios²,

Amsterdam³

et al. 2022

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Surgical instrument segmentation can be used in a range of computer assisted interventions and automation in surgical robotics. While deep learning architectures have rapidly advanced the robustness and performance of segmentation models, most are still reliant on supervision and large quantities of labelled data. In this paper, we present a novel method for surgical image generation that can fuse robotic instrument simulation and recent domain adaptation techniques to synthesize artificial surgical images to train surgical instrument segmentation models. We integrate attention modules into well established image generation pipelines and propose a novel cost function to support supervision from simulation frames in model training.We provide an extensive evaluation of our method in terms of segmentation performance along with a validation study on image quality using evaluation metrics. Additionally, we release a novel segmentation dataset from real surgeries that will be shared for research purposes. Both binary and semantic segmentation have been considered, and we show the capability of our synthetic images to train segmentation models compared with the latest methods from the literature.

show abstract

Long-Term Temporally Consistent Unpaired Video Translation from Simulated Surgical 3D Data

Cited by 19 publications

References 34 publications

Virtual Reality for Synergistic Surgical Training and Data Generation

Virtual Reality for Synergistic Surgical Training and Data Generation

Robot-Assisted Minimally Invasive Surgery—Surgical Robotics in the Data Age

SSIS-Seg: Simulation-Supervised Image Synthesis for Surgical Instrument Segmentation

Contact Info

Product

Resources

About