Patch-based adaptive weighting with segmentation and scale (PAWSS) for visual tracking in surgical video

Abstract

Tracking-by-detection algorithms are widely used for visual tracking, where the problem is treated as a classification task where an object model is updated over time using online learning techniques. In challenging conditions where an object undergoes deformation or scale variations, the update step is prone to include background information in the model appearance or to lack the ability to estimate the scale change, which degrades the performance of the classifier. In this paper, we incorporate a Patch-based Adaptive Weighting with Segmentation and Scale (PAWSS) tracking framework that tackles both the scale and background problems. A simple but effective colour-based segmentation model is used to suppress background information and multi-scale samples are extracted to enrich the training pool, which allows the tracker to handle both incremental and abrupt scale variations between frames. Experimentally, we evaluate our approach on the online tracking benchmark (OTB) dataset and Visual Object Tracking (VOT) challenge datasets. The results show that our approach outperforms recent state-of-the-art trackers, and it especially improves the successful rate score on the OTB dataset, while on the VOT datasets, PAWSS ranks among the top trackers while operating at real-time frame rates.

Towards real-time multiple surgical tool tracking

“…Each resulting cluster clst(i) is characterised by its center clst c (i), its radius r c (i) and its own members {f r t |f r t ∈ clst(i)}. Yang et al [20] considered all superpixels inside the active bounding box as the target area, which is a common assumption implying the region of interest/ bounding box contains very few background pixels [10]. When tracking whole tools, bounding boxes contain numerous background pixels, breaking the above assumption.…”

“…Recent approaches towards developing vision-based instrument tracking algorithms have focused on demonstrating feasibility in single object bounding box tracking [10,6,7]. A scale adaptive search strategy, as well as a probabilistic segmentation of background pixels were proposed as tools to increase tracking robustness during long sequences [10].…”

“…Recent approaches towards developing vision-based instrument tracking algorithms have focused on demonstrating feasibility in single object bounding box tracking [10,6,7]. A scale adaptive search strategy, as well as a probabilistic segmentation of background pixels were proposed as tools to increase tracking robustness during long sequences [10]. A recent comparison study of multiple vision-based single object trackers in minimally invasive surgery (MIS) data shows excellent performance on ex-vivo robotic sequences [6].…”

“…Moreover, such tool parametrisation allows for multi-instrument disambiguation. Overall, current methods rely on additional information such as robot kinematics and 3D CAD tool models [21] which might hinder clinical translation, they cannot handle tool occlusion [21,10,9,18] or recover temporal trajectories [21,9] and real-time capabilities are prohibitive [10,9].…”

“…Most importantly, the majority of proposed methods focus on tracking a point centered on the tool tip or the bounding box enclosing it [10,6,16]. While efficient, such approaches will fail due to occlusion during tool-tissue or tool-tool interactions.…”