Instrument State Recognition and Tracking for Effective Control of Robotized Laparoscopic Systems

Abstract

Surgical robots are an important component for delivering advanced paradigm shifting technology such as image guided surgery and navigation. However, for robotic systems to be readily adopted into the operating room they must be easy and convenient to control and facilitate a smooth surgical workflow. In minimally invasive surgery, the laparoscope may be held by a robot but controlling and moving the laparoscope remains challenging. It is disruptive to the workflow for the surgeon to put down the tools to move the robot in particular for solo surgery approaches. This paper proposes a novel approach for naturally controlling the robot mounted laparoscope's position by detecting a surgical grasping tool and recognizing if its state is open or close. This approach does not require markers or fiducials and uses a machine learning framework for tool and state recognition which exploits naturally occurring visual cues. Furthermore a virtual user interface on the laparoscopic image is proposed that uses the surgical tool as a pointing device to overcome common problems in depth perception. Instrument detection and state recognition are evaluated on in-vivo and ex-vivo porcine datasets. To demonstrate the practical surgical application and real time performance the system is validated in a simulated surgical environment. 

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

“…Tool detection generally is an intermediate step for tool tracking, the process of monitoring tool location over time (Du et al, 2016;Rieke et al, 2016a;Lee et al, 2017b;Zhao et al, 2017;Czajkowska et al, 2018;Ryu et al, 2018;Keller et al, 2018), and pose estimation, the process of inferring a 2-D pose (Rieke et al, 2016b;Kurmann et al, 2017;Alsheakhali et al, 2016b;Du et al, 2018;Wesierski and Jezierska, 2018) or a 3-D pose (Allan et al, 2018;Gessert et al, 2018) based on the location of tool elements. Tasks associated with tool detection also include velocity estimation (Marban et al, 2017) and instrument state recognition (Sahu et al, 2016a). All the above tasks are directly useful to the surgeon: they can be used for improved visualization, through augmented or mixed reality (Frikha et al, 2016 Nov-Dec;Bodenstedt et al, 2016 Feb-Mar;Lee et al, 2017b,a).…”

“…Various computer vision algorithms have been proposed to address these tasks. Until early 2017, tool detection relied heavily on handcrafted features, including Gabor filters (Czajkowska et al, 2018), Frangi filters (Agustinos and Voros, 2015;Chang et al, 2016), color-based features (Primus et al, 2016;Rieke et al, 2016a), histograms of oriented gradients (Rieke et al, 2016a;Czajkowska et al, 2018), SIFT features (Du et al, 2016), ORB features (Primus et al, 2016) and local binary patterns (Sahu et al, 2016a). For tool segmentation, similar features have been extracted within superpixels (Bodenstedt et al, 2016 Feb-Mar).…”

“…For tool segmentation, similar features have been extracted within superpixels (Bodenstedt et al, 2016 Feb-Mar). These features were processed either by a machine learning algorithm, such as a support vector machine (Primus et al, 2016;Wesierski and Jezierska, 2018), a random forest (Bodenstedt et al, 2016 Feb-Mar;Rieke et al, 2016a,b) or AdaBoost (Sahu et al, 2016a), or by a parametric model, such as a generalized Hough transform (Du et al, 2016;Frikha et al, 2016 Nov-Dec;Czajkowska et al, 2018) or a B-spline model (Chang et al, 2016). Note that template matching techniques have also been used to deal with articulated instruments (Ye et al, 2016;Wesierski and Jezierska, 2018).…”

Surgical Tools Detection Based on Modulated Anchoring Network in Laparoscopic Videos

“…With the widespread use of devices to record surgical procedures in minimally invasive surgery, automated analysis of surgical tools in videos has become a popular research area, mainly involving classification, segmentation, tracking, detection, and other directions. Unlike the earlier methods [6]- [11], rely on various handcrafted features, the existing approaches mainly use deep learning to extract more high-level features for surgical workflow recognition and tool detection. The traditional analysis of surgical phases is based on a number of statistical models, involving Conditional Random Fields [12]- [15], Hidden Markov Models [7], [16], [17], Hidden semi-Markov Models [18], [19], Linear Dynamical Systems [20] and so on.…”

LC-GAN: Image-to-image Translation Based on Generative Adversarial Network for Endoscopic Images

“…For example, Shan Lin, Randall A. Bly, Kris S. Moe and Blake Hannaford are with University of Washington, Seattle, WA 98195, USA. shanl3@uw.edu Fangbo Qin is with Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China qinfangbo2013@ia.ac.cn Yangming Li is with Rochester Institute of Technology, Rochester, NY 14623, USA Yangming.Li@rit.edu robots that handle the endoscope or laparoscope have been explored [8], [9], the applications of soft robotic devices in MIS have been studied to reduce patients' pain and damage [5]. Moreover, overlaying pre-and intra-operative imaging with surgical videos could improve surgeons' capabilities [10].…”