Unified Detection and Tracking of Instruments during Retinal Microsurgery

Abstract

Methods for tracking an object have generally fallen into two groups: tracking by detection and tracking through local optimization. The advantage of detection-based tracking is its ability to deal with target appearance and disappearance, but it does not naturally take advantage of target motion continuity during detection. The advantage of local optimization is efficiency and accuracy, but it requires additional algorithms to initialize tracking when the target is lost. To bridge these two approaches, we propose a framework for unified detection and tracking as a time-series Bayesian estimation problem. The basis of our approach is to treat both detection and tracking as a sequential entropy minimization problem, where the goal is to determine the parameters describing a target in each frame. To do this we integrate the Active Testing (AT) paradigm with Bayesian filtering, and this results in a framework capable of both detecting and tracking robustly in situations where the target object enters and leaves the field of view regularly. We demonstrate our approach on a retinal tool tracking problem and show through extensive experiments that our method provides an efficient and robust tracking solution.

Online tracking and retargeting with applications to optical biopsy in gastrointestinal endoscopic examinations

“…In surgical vision, detection and tracking have been topics of particular interest, which include both surgical instrument tracking or detection (Sznitman et al, 2012(Sznitman et al, , 2013(Sznitman et al, , 2014Reiter et al, 2014), and tissue tracking (Mountney et al, 2007;Mountney and Yang, 2008;Richa et al, 2011Richa et al, , 2012Giannarou et al, 2013). In this paper, we propose an Online Tracking and Retargeting (OTR) framework for optical biopsy.…”

Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method

“…Some other vision-based methods exploit the geometric constraints [8] and the gradientlike features [9,10], in order to identify the shaft of instrument, but fail to provide more accurate 3D positions of the instrument tip. Machine learning techniques [11][12][13][14][15][16][17][18][19] introduced into the instrument detection and tracking provide training of their discriminative classifiers/models according to the input visual features of the foreground (instrument tip or shaft). Edge pixel features [11] and fast corner features [12] are utilized to train the appearance models of surgical instrument based on the likelihood map.…”

“…region covariance (Covar) [14], scale invariant feature transform (SIFT) [15,16] and histogram of oriented gradients (HoG) [17], are used to establish the surgical instrument model in tracking and coupled with some traditional classifiers such as support vector machine (SVM), randomized tree (RT) and so on. The Bayesian sequential estimation was also applied to the surgical instrument tracking via the active testing model [18]. Some new metric measurements of image similarity, such as the sum of conditional variance (SCV) [19], have been advanced to improve the performance of instrument detection/tracking.…”

Human eye phantom for developing computer and robot-assisted epiretinal membrane peeling

“…Developments include 3D visualization systems and robust algorithms for tracking the relative motion of the retina and surgical tools [1-3] and for delineating blood vessels. These features would help provide real-time visual feedback during surgery that can additionally be used to generate virtual fixtures with assistive robots [4].…”

“…The development of a microsurgical assistant system for retinal surgery at the Johns Hopkins University has created the need for an eye phantom that combines several features such as ocular geometry, fundus appearance [1-3,5], and realistic ERM peeling behavior, including forces that simulate those encountered during surgery [21]. In this paper, we present an adaptable eye phantom as well as a quantitative assessment of the forces produced during delamination of membranes prepared from four candidate materials for simulating ERM peeling, a standard vitreoretinal microsurgical procedure.…”