Towards Unsupervised Learning for Instrument Segmentation in Robotic Surgery with Cycle-Consistent Adversarial Networks

Pakhomov, Daniil; Shen, Wei; Navab, Nassir

doi:10.1109/iros45743.2020.9340816

Cited by 13 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Marzullo et al (2021) use a pix2pix GAN to generate synthetic surgical images from rough segmentation mask of surgical instruments and tissues. In the context of robotic surgery, Pakhomov et al (2020) record synchronized surgical videos and kinematic joint values and then use the letter to generate synthetic annotations, projecting the estimated tool 3D shapes, obtained via forward kinematics, onto the image space; in order to take into account the possible inaccuracy of the tool model, the segmentation problem is formulated as unpaired image-to-image translation, using a cycle-GAN architecture. An alternative proposed solution to reduce the need for manual annotations is represented by semi-supervision using label propagation.…”

Section: Surgical Tool Segmentationmentioning

confidence: 99%

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

Sestini¹,

Rosa²,

Momi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic surgical instrument segmentation of endoscopic images is a crucial building block of many computer-assistance applications for minimally invasive surgery. So far, state-of-the-art approaches completely rely on the availability of a groundtruth supervision signal, obtained via manual annotation, thus expensive to collect at large scale. In this paper, we present FUN-SIS, a Fully-UNsupervised approach for binary Surgical Instrument Segmentation. FUN-SIS trains a per-frame segmentation model on completely unlabelled endoscopic videos, by solely relying on implicit motion information and instrument shapepriors. We define shape-priors as realistic segmentation masks of the instruments, not necessarily coming from the same dataset/domain as the videos. The shape-priors can be collected in various and convenient ways, such as recycling existing annotations from other datasets. We leverage them as part of a novel generative-adversarial approach, allowing to perform unsupervised instrument segmentation of optical-flow images during training. We then use the obtained instrument masks as pseudo-labels in order to train a per-frame segmentation model; to this aim, we develop a learning-from-noisy-labels architecture, designed to extract a clean supervision signal from these pseudo-labels, leveraging their peculiar noise properties. We validate the proposed contributions on three surgical datasets, including the MICCAI 2017 EndoVis Robotic Instrument Segmentation Challenge dataset. The obtained fully-unsupervised results for surgical instrument segmentation are almost on par with the ones of fully-supervised state-of-the-art approaches. This suggests the tremendous potential of the proposed method to leverage the great amount of unlabelled data produced in the context of minimally invasive surgery.

show abstract

Section: Surgical Tool Segmentationmentioning

confidence: 99%

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

Sestini¹,

Rosa²,

Momi³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, attempts to do so have shown many drawbacks besides a limited accuracy [10], [11]. In addition to the difficulties created by the added complexity due to the workflow alteration, additional sensors or tool modifications have to be able to overcome the harsh conditions of the instrument sterilization process [12].…”

Section: Foregroundmentioning

confidence: 99%

“…Υ is proportional to the number of optimization steps or training iterations selected. We refer to the learning strategy in (12) as mix-blend learning (see fig. 3 and supplementary material fig.…”

Section: A Semi-synthetic Learningmentioning

confidence: 99%

Image Compositing for Segmentation of Surgical Tools Without Manual Annotations

García-Peraza-Herrera

Fidon

D'Ettorre

et al. 2021

IEEE Trans. Med. Imaging

View full text Add to dashboard Cite

Producing manual, pixel-accurate, image segmentation labels is tedious and time-consuming. This is often a rate-limiting factor when large amounts of labeled images are required, such as for training deep convolutional networks for instrument-background segmentation in surgical scenes. No large datasets comparable to industry standards in the computer vision community are available for this task. To circumvent this problem, we propose to automate the creation of a realistic training dataset by exploiting techniques stemming from special effects and harnessing them to target training performance rather than visual appeal. Foreground data is captured by placing sample surgical instruments over a chroma key (a.k.a. green screen) in a controlled environment, thereby making extraction of the relevant image segment straightforward. Multiple lighting conditions and viewpoints can be captured and introduced in the simulation by moving the instruments and camera and modulating the light source. Background data is captured by collecting videos that do not contain instruments. In the absence of pre-existing instrument-free background videos, minimal labeling effort is required, just to select frames that do not contain surgical instruments from videos of surgical interventions freely available online. We compare different methods to blend instruments over tissue and propose a novel data augmentation approach that takes advantage of the plurality of options. We show that by training a vanilla U-Net on semi-synthetic data only and applying a simple post-processing, we are able to match the results of the same network trained on a publicly available manually labeled real dataset.

show abstract

“…learning the mapping from cadaveric to in vivo images [14]. The in vivo images are unlabeled, while the cadaveric images are from a labeled dataset and can be obtained by rendering CAD models of each tool [15]. The implementation of CycleGAN works for surgical tool segmentation with unlabeled data for live images.…”

Section: Related Workmentioning

confidence: 99%

Surgical Tool Segmentation Using Generative Adversarial Networks With Unpaired Training Data

Zhang

Rosa

Nageotte

2021

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Surgical tool segmentation is a challenging and crucial task for computer and robot-assisted surgery. Supervised learning approaches have shown great success for this task. However, they need a large number of paired training data. Based on Generative Adversarial Networks (GAN), unpaired image-toimage translation (I2I) techniques (like CycleGAN and dualGAN) have been proposed to avoid the requirement of paired data and have been employed for surgical tool segmentation. The unpaired I2I methods avoid annotating images for domain changes. Instead of using them directly for the segmentation task, we propose new GAN-based methods for unpaired I2I by embedding a specific constraint for segmentation, namely each pixel of input image belongs to either background or surgical tool. Our methods simplify the architectures of existing unpaired I2I with a reduced number of generators and discriminators. Compared with dualGAN, the proposed strategies have a faster training process without reducing the accuracy of the segmentation. Besides, we show that, using textured tool images as annotated samples to train discriminators, unpaired I2I (including our methods) can achieve simultaneous tool image segmentation and repair (such as reflection removal and tool inpainting). The proposed strategies are validated for image segmentation of a flexible tool and for in vivo images from the EndoVis dataset.

show abstract

Towards Unsupervised Learning for Instrument Segmentation in Robotic Surgery with Cycle-Consistent Adversarial Networks

Cited by 13 publications

References 16 publications

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

Image Compositing for Segmentation of Surgical Tools Without Manual Annotations

Surgical Tool Segmentation Using Generative Adversarial Networks With Unpaired Training Data

Contact Info

Product

Resources

About