Simulation-to-real domain adaptation with teacher–student learning for endoscopic instrument segmentation

Abstract

Purpose Segmentation of surgical instruments in endoscopic video streams is essential for automated surgical scene understanding and process modeling. However, relying on fully supervised deep learning for this task is challenging because manual annotation occupies valuable time of the clinical experts. Methods We introduce a teacher–student learning approach that learns jointly from annotated simulation data and unlabeled real data to tackle the challenges in simulation-to-real unsupervised domain adaptation for endoscopic image segmentation. Results Empirical results on three datasets highlight the effectiveness of the proposed framework over current approaches for the endoscopic instrument segmentation task. Additionally, we provide analysis of major factors affecting the performance on all datasets to highlight the strengths and failure modes of our approach. Conclusions We show that our proposed approach can successfully exploit the unlabeled real endoscopic video frames and improve generalization performance over pure simulation-based training and the previous state-of-the-art. This takes us one step closer to effective segmentation of surgical instrument in the annotation scarce setting.

SSIS-Seg: Simulation-Supervised Image Synthesis for Surgical Instrument Segmentation

“…Starting from binary segmentation, the model trained using the proposed data generation workflow outperformed all the other models, achieving mean IoU scores of 85.26%, 93.02% and 57.72% on UCL, MICCAI '17 and RARP45 datasets respectively. Here, we noticed a more consistent divergence compared to CoSegGAN and Endo-sim2real [23], [24] and differences around 2-3% with Colleoni et al and Ozawa et al (on MICCAI '17). Following experiments carried in [9], we believe that these results can be explained and also enforce the idea that a better image quality leads DL models to enhanced segmentation capabilities.…”

“…Surprisingly, Endo-sim2real and Endo-sim2real (v2) showed performances far below the ones they achieved when they have been proposed. A potential explanation can lie in the simulation instruments considered in our work that greatly differ from the more realistic ones used in [23], [24], thus breaking the consistency learning framework employed by these two methods. All segmentation models trained on simulation data (No I2I) were not able to reach more than 39.52% IoU on UCL dataset while decreasing to ∼5% and 0% when testing on MICCAI '17 and RARP45, thus proving one more time the efficacy of applying I2I techniques to support supervised training.…”

“…Similar approaches [22] attempted to expand the range of different surgical instruments within the image, but realism is lost in the translation. Alternative domain adaptation methods based on consistency losses and student-teacher learning [23], [24] have also been proposed for surgical instrument segmentation, while [25] explores surgical tool segmentation aided by unpaired adversarial training.…”

“…In this fashion, we fitted binary and semantic segmentation models on frames produced by all the studies considered in Sec.IV-B.2 (using synthetic frames both for training and validation) while relying on real data only for testing purposes. Additionally, we introduced Endo-sim2real [23] and its updated version (Endo-sim2real (v2) [24]) in our study as they perform unsupervised surgical tool segmentation. We also performed additional training over simulation data without I2I and over real data (when possible) to provide lower and higher accuracy boundaries for each category.…”

Surgical Tool Datasets for Machine Learning Research: A Survey

“…This method achieved a mean Dice score of 87.41 (94.35) on the ROBUST-MIS dataset. Sahu et al (2021) used a teacher-student learning approach that learned from annotated simulation data and unlabeled real data. They redesigned their Endo-Sim2Real framework based on a teacher-student approach, and used a TerNaus11 as the backbone segmentation model.…”