Gesture Recognition in Robotic Surgery: A Review

Abstract

Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with combinations of the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. Results: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. Conclusion: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. Significance: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.

A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

“…The presented approach focuses on the task state estimation of a known task series. This approach could be extended by surgical gesture recognition inferring which task the surgeon aims to complete ( van Amsterdam et al, 2021 ). This would allow for more flexible task series.…”

“…However, VFs have to be parameterized according to the context of the task at hand in order to provide reasonable assistance. Due to the unstructured dynamic environment inside the patient’s body, automatic perception of the procedural context in a real surgery remains challenging ( van Amsterdam et al, 2021 ). Surgical training, in turn, offers a well-defined environment.…”

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

“…In [176], the legal implications of using AI for automation in surgical practice are discussed, while virtual and augmented reality in robotic surgery are reviewed in [51]. A recent review of gesture analysis in surgical robotics summarized the state of the art in this field [266].…”

Rethinking Autonomous Surgery: Focusing on Enhancement over Autonomy

“…In this case, kinematic measurements can be obtained, for example, via wireless sensors [34], electromagnetic sensors [35], and optical and camera trackers [36]. Nonsystematic review [15][16][17] Research [18][19][20][21] Validation of measurements [18,20,21] Crossover trial [19] Baseline data collections and wearable sensors are always required Vision-based Nonsystematic review [22][23][24][25] Systematic review [26] Research [27][28][29][30][31] Technical validation [27][28][29][30][31] Persisting challenges in image segmentation, black-box algorithms do not easily translate to training strategies Motion-based Nonsystematic review [32] Research [33][34][35][36] Validation of measurements [33] Validation of assessment [34][35][36] Augmenting robot sensing capabilities…”

“…surgeon (or superior) remains distant, significant technical progress continues to be made, piecewise, in allowing quantitative, vision-based feedback for guiding and informing robotic surgery. Video-based methods have been proposed for a variety of relevant objectives [22], including characterization of tool articulation and kinematics [27,28], phase and step recognition in surgical procedures [29], classification of action, gestures, and tasks [26], and assessment of surgical skill [23,30,31]. Notably, at the heart of state-of-the-art approaches to surgical video analysis is deep learning (DL), a subfield of machine learning involving models that can automatically learn multiple layers of data representation to capture increasingly complex patterns in a hierarchical fashion [24].…”