Deep Residual Learning for Instrument Segmentation in Robotic Surgery

Pakhomov, Daniil; Premachandran, Vittal; Allan, Max; Azizian, Mahdi; Navab, Nassir

doi:10.48550/arxiv.1703.08580

Cited by 12 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Luis et al [9] presented a network based on FCN and optic flow to solve problems such as occlusion and deformation of surgical instruments. Another work [10] used the residual network with dilated convolutions to segment surgical instruments. However, most of these work mainly focuses on the improvement of segmentation accuracy while fails to segment surgical instruments in real-time.…”

Section: A Semantic Segmentation Of Surgical Instrumentsmentioning

confidence: 99%

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Bian

Hou

et al. 2019

Preprint

View full text Add to dashboard Cite

Real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, realtime segmentation of surgical instruments using current deep learning models is still a challenging task due to the high computational costs and slow inference speed. In this paper, an attention-guided lightweight network (LWANet), is proposed to segment surgical instruments in real-time. LWANet adopts the encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2 and the decoder consists of depth-wise separable convolution, attention fusion block, and transposed convolution. Depth-wise separable convolution is used as the basic unit to construct the decoder, which can reduce the model size and computational costs. Attention fusion block captures global context and encodes semantic dependencies between channels to emphasize target regions, contributing to locating the surgical instrument. Transposed convolution is performed to upsample the feature map for acquiring refined edges. LWANet can segment surgical instruments in real-time, taking few computational costs. Based on 960×544 inputs, its inference speed can reach 39 fps with only 3.39 GFLOPs. Also, it has a small model size and the number of parameters is only 2.06 M. The proposed network is evaluated on two datasets. It achieves state-of-the-art performance 94.10% mean IOU on Cata7 and obtains a new record on EndoVis 2017 with 4.10% increase on mean mIOU.

show abstract

Section: A Semantic Segmentation Of Surgical Instrumentsmentioning

confidence: 99%

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Bian

Hou

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Despite development in surgical technology, sometimes surgeons may lose surgical workflow due to weak tactile feedback or system impairment. Moreover, a compound, surgical scenario with smoke, body fluid, blood, adverse lighting condition, and partial occlusion creates additional challenges in image cognition [2]. Therefore, concurrent detection and segmentation of instruments could enhance surgical outcomes and assist novice surgeons with real-time objective feedback, do skill assessment, and analyze tool movements in the surgical workflow.…”

Section: Introductionmentioning

confidence: 99%

“…Instrument segmentation of binary, parts and type-wise is done by LinkNet [8] with Jaccard index based loss function [9]. Subsequently, ToolNet [10] with holistically-nested architecture, and dilated residual network with multi-scaled inputs [2] are utilized to segment the surgical tool. Nonetheless, these works lack focusing on the real-time application in terms of both speed and accuracy.…”

Section: Introductionmentioning

confidence: 99%

AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery

Islam

Ren

2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Surgical scene understanding and multi-tasking learning are crucial for image-guided robotic surgery. Training a real-time robotic system for the detection and segmentation of high-resolution images provides a challenging problem with the limited computational resource. The perception drawn can be applied in effective real-time feedback, surgical skill assessment, and human-robot collaborative surgeries to enhance surgical outcomes. For this purpose, we develop a novel end-to-end trainable real-time Multi-Task Learning (MTL) model with weight-shared encoder and task-aware detection and segmentation decoders. Optimization of multiple tasks at the same convergence point is vital and presents a complex problem. Thus, we propose an asynchronous task-aware optimization (ATO) technique to calculate task-oriented gradients and train the decoders independently. Moreover, MTL models are always computationally expensive, which hinder real-time applications. To address this challenge, we introduce a global attention dynamic pruning (GADP) by removing less significant and sparse parameters. We further design a skip squeeze and excitation (SE) module, which suppresses weak features, excites significant features and performs dynamic spatial and channelwise feature re-calibration. Validating on the robotic instrument segmentation dataset of MICCAI endoscopic vision challenge, our model significantly outperforms state-of-the-art segmentation and detection models, including best-performed models in the challenge.

show abstract

“…Multi-class (both instrument part and type) tool segmentation was first proposed by Shvets et al [13], and Pakhomov et al [14] and achieved promising results. They modified the classic U-Net model [7] that relies on the transposed convolution or deconvolution, in a similar, yet opposite fashion to the convolutional layers.…”

Section: Introductionmentioning

confidence: 99%

U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument

Hasan¹,

Linte²

2019

Preprint

View full text Add to dashboard Cite

Conventional therapy approaches limit surgeons' dexterity control due to limited field-of-view. With the advent of robot-assisted surgery, there has been a paradigm shift in medical technology for minimally invasive surgery. However, it is very challenging to track the position of the surgical instruments in a surgical scene, and accurate detection & identification of surgical tools is paramount. Deep learning-based semantic segmentation in frames of surgery videos has the potential to facilitate this task. In this work, we modify the U-Net architecture named U-NetPlus, by introducing a pre-trained encoder and re-design the decoder part, by replacing the transposed convolution operation with an upsampling operation based on nearest-neighbor (NN) interpolation. To further improve performance, we also employ a very fast and flexible data augmentation technique. We trained the framework on 8 × 225 frame sequences of robotic surgical videos, available through the MICCAI 2017 EndoVis Challenge dataset and tested it on 8 × 75 frame and 2 × 300 frame videos. Using our U-NetPlus architecture, we report a 90.20% DICE for binary segmentation, 76.26% DICE for instrument part segmentation, and 46.07% for instrument type (i.e., all instruments) segmentation, outperforming the results of previous techniques implemented and tested on these data.

show abstract

Deep Residual Learning for Instrument Segmentation in Robotic Surgery

Cited by 12 publications

References 0 publications

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

Attention-Guided Lightweight Network for Real-Time Segmentation of Robotic Surgical Instruments

AP-MTL: Attention Pruned Multi-task Learning Model for Real-time Instrument Detection and Segmentation in Robot-assisted Surgery

U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument

Contact Info

Product

Resources

About