Mask then classify: multi-instance segmentation for surgical instruments

Abstract

Purpose The detection and segmentation of surgical instruments has been a vital step for many applications in minimally invasive surgical robotics. Previously, the problem was tackled from a semantic segmentation perspective, yet these methods fail to provide good segmentation maps of instrument types and do not contain any information on the instance affiliation of each pixel. We propose to overcome this limitation by using a novel instance segmentation method which first masks instruments and then classifies them into their respective type. Methods We introduce a novel method for instance segmentation where a pixel-wise mask of each instance is found prior to classification. An encoder–decoder network is used to extract instrument instances, which are then separately classified using the features of the previous stages. Furthermore, we present a method to incorporate instrument priors from surgical robots. Results Experiments are performed on the robotic instrument segmentation dataset of the 2017 endoscopic vision challenge. We perform a fourfold cross-validation and show an improvement of over 18% to the previous state-of-the-art. Furthermore, we perform an ablation study which highlights the importance of certain design choices and observe an increase of 10% over semantic segmentation methods. Conclusions We have presented a novel instance segmentation method for surgical instruments which outperforms previous semantic segmentation-based methods. Our method further provides a more informative output of instance level information, while retaining a precise segmentation mask. Finally, we have shown that robotic instrument priors can be used to further increase the performance.

The Advances in Computer Vision That Are Enabling More Autonomous Actions in Surgery: A Systematic Review of the Literature

“…As opposed to standard laparoscopy, robotic-assisted surgery that uses complete robotic surgical systems has the capacity to record and interpret data regarding instrument location into DL algorithms. These types of data are referred to as instrument priors can enhance a robots ability to accurately identify robotic surgical instruments and even their respective parts [31]. As instrument segmentation has evolved, researchers have shown that algorithms can be developed that can even correctly interpret datasets that are publicly available and not only datasets obtained locally under controlled environments.…”

Surgical Tool Datasets for Machine Learning Research: A Survey

“…They evaluated their architecture on the EndoVis17 and an in-house hysterectomy dataset. Kurmann et al (2021) proposed a encoder-decoder network for segmentation and classification of surgical instruments in endoscopic images. Their "segment first, classify last" approach used a shared encoder, two decoders for instance segmentation, and a classifier for instance classification, and it provided good results on the EndoVis 2017 dataset.…”

FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation

“…Ni et al (2020) propose a bilinear attention network with an adaptive receptive field to tackle the challenges of scale and illumination inter-frames variability. Kurmann et al (2021) propose an alternative to standard semantic segmentation, first extracting instrument instances and then independently classifying them, reaching state-of-the-art results for this task. Despite the good results obtained by fullysupervised methods, their application is inherently limited by the need for manual annotations, which prevents their scalability.…”