ToolNet: Holistically-nested real-time segmentation of robotic surgical tools

García-Peraza-Herrera, Luis C.; Li, Wenqi; Fidon, Lucas; Gruijthuijsen, Caspar; Devreker, Alain; Attilakos, George; Deprest, Jan; Poorten, Emmanuel Vander; Stoyanov, Danail; Vercauteren, Tom; Ourselin, Sébastien

doi:10.1109/iros.2017.8206462

Cited by 124 publications

(82 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Later, convolutional neural network (CNN) based methods have demonstrated new state-of-the-art on instrument segmentation. The ToolNet [5] uses a holistically-nested fully convolutional network, imposing multi-scale constraint of predictions. Laina et al [9] propose a multi-task CNN to concurrently regress the segmentation and localization.…”

Section: Introductionmentioning

confidence: 99%

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Jin

Cheng

Dou

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Automatic instrument segmentation in video is an essentially fundamental yet challenging problem for robot-assisted minimally invasive surgery. In this paper, we propose a novel framework to leverage instrument motion information, by incorporating a derived temporal prior to an attention pyramid network for accurate segmentation. Our inferred prior can provide reliable indication of the instrument location and shape, which is propagated from the previous frame to the current frame according to inter-frame motion flow. This prior is injected to the middle of an encoder-decoder segmentation network as an initialization of a pyramid of attention modules, to explicitly guide segmentation output from coarse to fine. In this way, the temporal dynamics and the attention network can effectively complement and benefit each other. As additional usage, our temporal prior enables semi-supervised learning with periodically unlabeled video frames, simply by reverse execution. We extensively validate our method on the public 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset with three different tasks. Our method consistently exceeds the state-of-the-art results across all three tasks by a large margin. Our semi-supervised variant also demonstrates a promising potential for reducing annotation cost in the clinical practice.

show abstract

Section: Introductionmentioning

confidence: 99%

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Jin

Cheng

Dou

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In the literature, surgical tool localization has traditionally been approached with fully supervised methods [1], with the most recent localization and segmentation methods relying on deep learning [4,8,10,11,13]. However, training fully supervised approaches require the data to be fully annotated with spatial information, which is tedious and expensive.…”

Section: Introductionmentioning

confidence: 99%

Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos

Vardazaryan

Mutter

Marescaux

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Surgical tool localization is an essential task for the automatic analysis of endoscopic videos. In the literature, existing methods for tool localization, tracking and segmentation require training data that is fully annotated, thereby limiting the size of the datasets that can be used and the generalization of the approaches. In this work, we propose to circumvent the lack of annotated data with weak supervision. We propose a deep architecture, trained solely on image level annotations, that can be used for both tool presence detection and localization in surgical videos. Our architecture relies on a fully convolutional neural network, trained end-to-end, enabling us to localize surgical tools without explicit spatial annotations. We demonstrate the benefits of our approach on a large public dataset, Cholec80, which is fully annotated with binary tool presence information and of which 5 videos have been fully annotated with bounding boxes and tool centers for the evaluation.

show abstract

“…However both surgical robotics and robotic imaging will play increasingly crucial roles in the years to come. Machine learning is demonstrating convincing results in real-time tool tracking [118], [172]- [174]. This for example enables automatic positioning of intra-operative OCT imaging planes within surgical microscopy for ophthalmic surgery [119], [175].…”

Section: Discussionmentioning

confidence: 97%

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

et al. 2020

Self Cite

View full text Add to dashboard Cite

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.Index Terms-Artificial intelligence, computer assisted interventions, interventional workflow, intra-operative imaging, surgical planning, data fusion, surgical scene understanding, contextaware user interface, machine and deep learning, surgical data science

show abstract

ToolNet: Holistically-nested real-time segmentation of robotic surgical tools

Cited by 124 publications

References 22 publications

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Incorporating Temporal Prior from Motion Flow for Instrument Segmentation in Minimally Invasive Surgery Video

Weakly-Supervised Learning for Tool Localization in Laparoscopic Videos

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

Contact Info

Product

Resources

About