Vision-based and marker-less surgical tool detection and tracking: a review of the literature

Bouget, David; Allan, Max; Stoyanov, Danail; Jannin, Pierre

doi:10.1016/j.media.2016.09.003

Cited by 215 publications

(145 citation statements)

References 110 publications

Supporting

Mentioning

144

Contrasting

Order By: Relevance

“…With the advancements in medicine and information technologies, the operating room (OR) has undergone tremendous transformations evolving into a highly complicated and technologically rich environment (Cleary & Kinsella (2005); James et al (2007); Lalys & Jannin (2014); Bouget et al (2017)). These transformations allow the execution of more complex procedures and also increase the amount of information available in modern OR.…”

Section: Introductionmentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

145

View full text Add to dashboard Cite

Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both lowlevel feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition). The code can be found on our project website.

show abstract

Section: Introductionmentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

145

View full text Add to dashboard Cite

show abstract

“…Recently, a large body of work has targeted the detection and segmentation of surgical instruments [114]. Deep learning methods have been proposed for both bounding box or articulated tool detection [115]- [117] and for pixel-based tool segmentation [118], [119].…”

Section: B Understanding Image Semanticsmentioning

confidence: 99%

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

et al. 2020

View full text Add to dashboard Cite

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.Index Terms-Artificial intelligence, computer assisted interventions, interventional workflow, intra-operative imaging, surgical planning, data fusion, surgical scene understanding, contextaware user interface, machine and deep learning, surgical data science

show abstract

“…3) Endoscopic Surgical Tool Tracking and Control: A recent literature survey by Bouget et al [20] gave a detailed summary of image-based surgical tool detection. Markerless with tracking algorithms [21], [22], [23] requires features which can be learned [24], [25], generated via template matching [26], or hand-crafted [27].…”

Section: Related Workmentioning

confidence: 99%

“…2) Features Detection and Camera Projections: Algorithms developed in previous literature can be utilized to detect features from the endoscopic image data on the surgical tool to update the estimation for the particle filter [20]. However, colored markers were drawn on the surgical tool using a paint pen for the sake of simplicity.…”

Section: A Surgical Tool Trackingmentioning

confidence: 99%

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics

Richter

et al. 2020

IEEE Robot. Autom. Lett.

View full text Add to dashboard Cite

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modelled robotic systems in conjunction with multiple sensors. However, application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping of a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical R System in real-time with an endeffector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed autonomous soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research † .

show abstract

Vision-based and marker-less surgical tool detection and tracking: a review of the literature

Cited by 215 publications

References 110 publications

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics

Contact Info

Product

Resources

About