Data-Driven Visual Tracking in Retinal Microsurgery

Abstract

Abstract. In the context of retinal microsurgery, visual tracking of instruments is a key component of robotics assistance. The difficulty of the task and major reason why most existing strategies fail on in-vivo image sequences lies in the fact that complex and severe changes in instrument appearance are challenging to model. This paper introduces a novel approach, that is both data-driven and complementary to existing tracking techniques. In particular, we show how to learn and integrate an accurate detector with a simple gradient-based tracker within a robust pipeline which runs at framerate. In addition, we present a fully annotated dataset of retinal instruments in in-vivo surgeries, which we use to quantitatively validate our approach. We also demonstrate an application of our method in a laparascopy image sequence.

Fast Part-Based Classification for Instrument Detection in Minimally Invasive Surgery

“…1(c) illustrates the evolution of stage responses of each instrument part after evaluating the patch from (a) using a learned classifier. As in [10], these features can easily be extended to be rotationally invariant and generalize well to variations across image sequences.…”

“…Weighted averaging [10] on the response scores is then performed for each class to estimate the position of the different parts of the instrument, allowing the instrument center and orientation to be extracted from the part labels. In our experiments, we ran RANSAC with 500 sampling rounds and let inliers be points within 24 pixels of the model, i.e.…”

“…Among the many approaches proposed in the last twenty years [1,2,6,7,8,9], recent detection-based schemes that rely on building statistical classifiers to evaluate the presence of the instrument appear to be the most promising for in-vivo detection and tracking [4,5,10]. Within this last category of methods, Reiter et al [5] combined a multiclass Random Forest (RF) [11] labelling approach with robot kinematic information to estimate the instrument 3D pose.…”

“…In [4], RFs were also used to handle instrument-background classification, giving way to instrument segmentations and the 3D pose. Alternatively, [10] combined template tracking and binary classification to provide 2D instrument positions in retinal microsurgery.…”

“…As a result it is performed at a single predefined image scale and using only a limited number of RF trees. Furthermore, classification in [5] is only performed on pre-segmented pixels of the instrument, while [10] strongly relies on template tracking to apply a binary classifier on a small window of candidates. Consequently, to attain framerates of 1-5 fps, these methods must ignore structural information about the instrument, such as the position of the shaft, grippers or tip, ultimately impacting detection accuracy.…”