Surgical Tool Segmentation Using Generative Adversarial Networks With Unpaired Training Data

Abstract

Surgical tool segmentation is a challenging and crucial task for computer and robot-assisted surgery. Supervised learning approaches have shown great success for this task. However, they need a large number of paired training data. Based on Generative Adversarial Networks (GAN), unpaired image-toimage translation (I2I) techniques (like CycleGAN and dualGAN) have been proposed to avoid the requirement of paired data and have been employed for surgical tool segmentation. The unpaired I2I methods avoid annotating images for domain changes. Instead of using them directly for the segmentation task, we propose new GAN-based methods for unpaired I2I by embedding a specific constraint for segmentation, namely each pixel of input image belongs to either background or surgical tool. Our methods simplify the architectures of existing unpaired I2I with a reduced number of generators and discriminators. Compared with dualGAN, the proposed strategies have a faster training process without reducing the accuracy of the segmentation. Besides, we show that, using textured tool images as annotated samples to train discriminators, unpaired I2I (including our methods) can achieve simultaneous tool image segmentation and repair (such as reflection removal and tool inpainting). The proposed strategies are validated for image segmentation of a flexible tool and for in vivo images from the EndoVis dataset.

SSIS-Seg: Simulation-Supervised Image Synthesis for Surgical Instrument Segmentation

“…Additionally, the potential of surgical simulators could be exploited to produce different sources of labels, e.g., surgical tool pose or depth maps. This is not possible in other methods [21], [25] that are confined to surgical image segmentation. Moreover, given the availability of CAD models, a more comprehensive range of surgical instruments could be considered.…”

“…Similar approaches [22] attempted to expand the range of different surgical instruments within the image, but realism is lost in the translation. Alternative domain adaptation methods based on consistency losses and student-teacher learning [23], [24] have also been proposed for surgical instrument segmentation, while [25] explores surgical tool segmentation aided by unpaired adversarial training.…”

Evolution of Surgical Robot Systems Enhanced by Artificial Intelligence: A Review

“…Additionally, some researchers have achieved commendable results when employing unsupervised learning methods for instrument segmentation to avoid difficulties when providing paired datasets for supervised learning. [124] In pathological classification, AI has shown advantages in lesion recognition and classification. It assists in the rapid and accurate diagnosis of diseases and serves as a standardized and automated subprocess to enhance the integrity of SRS workflows.…”

“…Additionally, some researchers have achieved commendable results when employing unsupervised learning methods for instrument segmentation to avoid difficulties when providing paired datasets for supervised learning. [ 124 ]…”

Surgical Tool Datasets for Machine Learning Research: A Survey

“…This was tested on the EndoVis 2017 dataset. Zhang et al (2021b) proposed a GAN-based method for unpaired imageto-image translation (I2I), and used it for surgical tool image segmentation and repair. They tested this on three endoscopic surgery datasets and on the EndoVis17 dataset.…”