An Improved Model for Segmentation and Recognition of Fine-Grained Activities with Application to Surgical Training Tasks

Abstract

Automated segmentation and recognition of fine-grained activities is important for enabling new applications in industrial automation, human-robot collaboration, and surgical training. Many existing approaches to activity recognition assume that a video has already been segmented and perform classification using an abstract representation based on spatio-temporal features. While some approaches perform joint activity segmentation and recognition, they typically suffer from a poor modeling of the transitions between actions and a representation that does not incorporate contextual information about the scene. In this paper, we propose a model for action segmentation and recognition that improves upon existing work in two directions. First, we develop a variation of the Skip-Chain Conditional Random Field that captures long-range state transitions between actions by using higher-order temporal relationships. Second, we argue that in constrained environments, where the relevant set of objects is known, it is better to develop features using high-level object relationships that have semantic meaning instead of relying on abstract features. We apply our approach to a set of tasks common for training in robotic surgery: suturing, knot tying, and needle passing, and show that our method increases micro and macro accuracy by 18.46% and 44.13% relative to the state of the art on a widely used robotic surgery dataset.

Unsupervised Trajectory Segmentation for Surgical Gesture Recognition in Robotic Training

“…Their dual objective was to segment and recognize surgical gestures based on the temporal model of a surgical task. In a similar vein, more recent works [16]- [18] have proposed combining video and kinematic data, and have proved that mixing information from multiple modalities strongly improves gesture recognition capacities when building a temporal model. Despite such improvements, all these works relied on the assumption that the training sessions already comprised a breakdown of specific, recognizable gestures, which requires significant preprocessing input from experts.…”

A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery

“…Later work extended this to hidden Markov models (HMMs) learned from kinematic (hand-movement) data [19]–[21], [24], [25] using simple statistical models. Recent work [22]–[25], [29]–[33] extended basic HMM models in a variety of ways, and introduced conditional random fields as an alternative discriminative approach. These approaches all model each gesture as one or more latent variables.…”

Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization

“…Recent works have also explored the use of learning techniques to infer surgeme transitions from demonstration data [16,24]. Many of the FSRS procedures, including MTS, are decomposable into long sequences of simpler sub-tasks.…”

“…This paper builds on prior work in optimization-based planning [4,27], sub-task level segmentation of demonstrations [15,16], gripper mounted interchangeable tools [19], and building robust finite state machines [21]. We are not aware of any system that can perform autonomous multithrow suturing.…”