Computer Vision in the Surgical Operating Room

Chadebecq, François; Vasconcelos, Francisco; Mazomenos, Evangelos B.; Stoyanov, Danail

doi:10.1159/000511934

Cited by 40 publications

(33 citation statements)

References 52 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Surgical specialisations like gynecology and urology have driven the adoption of RMIS, but diffusion of the robotic approach is increasing as hospitals become equipped with surgical robotic systems, surgeons trained to use them and both the capital and running costs reduce [2]. In addition to better surgical instrumentation, surgical robots can potentially be platforms for enabling data driven solutions for better OR capabilities [3]. Besides digital video, surgical robots can capture quantitative instrument motion trajectories during surgical interventions enabling analysis of surgical activity that is not possible with traditional instrumentation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Gesture Recognition in Robotic Surgery: A Review

Amsterdam

Clarkson²,

Stoyanov³

2021

IEEE Trans. Biomed. Eng.

Self Cite

View full text Add to dashboard Cite

Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with combinations of the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. Results: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. Conclusion: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. Significance: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.

show abstract

Section: Introductionmentioning

confidence: 99%

“…beatrice.amsterdam.18@ucl.ac.uk surgeons trained to use them and both the capital and running costs reduce [2]. In addition to better surgical instrumentation, surgical robots can potentially be platforms for enabling data driven solutions for better OR capabilities [3].…”

Section: Introductionmentioning

confidence: 99%

Gesture Recognition in Robotic Surgery: A Review

Amsterdam

Clarkson²,

Stoyanov³

2021

IEEE Trans. Biomed. Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…2 shows a diagram of our modular architecture. The proposed framework is built upon a single convolutional encoder that extracts deep feature representation from input images, at 5 different scales S i , i ∈ (2,3,4,5,6). Each task has a dedicated convolutional sub-network (head) that makes predictions based on multi-scale inputs from the common feature encoder.…”

Section: Methodsmentioning

confidence: 99%

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

Psychogyios

Mazomenos

Vasconcelos

et al. 2022

IEEE Trans. Med. Imaging

Self Cite

View full text Add to dashboard Cite

Reconstructing the 3D geometry of the surgical site and detecting instruments within it are important tasks for surgical navigation systems and robotic surgery automation. Traditional approaches treat each problem in isolation and do not account for the intrinsic relationship between segmentation and stereo matching. In this paper, we present a learning-based framework that jointly estimates disparity and binary tool segmentation masks. The core component of our architecture is a shared feature encoder which allows strong interaction between the aforementioned tasks. Experimentally, we train two variants of our network with different capacities and explore different training schemes including both multi-task and single-task learning. Our results show that supervising the segmentation task improves our network's disparity estimation accuracy. We demonstrate a domain adaptation scheme where we supervise the segmentation task with monocular data and achieve domain adaptation of the adjacent disparity task, reducing disparity End-Point-Error and depth mean absolute error by 77.73% and 61.73% respectively compared to the pre-trained baseline model. Our best overall multi-task model, trained with both disparity and segmentation data in subsequent phases, achieves 89.15% mean Intersection-over-Union in RIS and 3.18 millimetre depth mean absolute error in SCARED test sets. Our proposed multi-task architecture is real-time, able to process (1280x1024) stereo input and simultaneously estimate disparity maps and segmentation masks at 22 frames per second. The model code and pre-trained models are made available: https://github.com/dimitrisPs/msdesis

show abstract

“…S URGICAL robots are now an established part of clinical practice, particularly in minimally invasive surgery, where surgeons especially benefit from the enhanced instrumentation, visualisation and ergonomics during the procedure [1]. In addition to advantages for the patient and clinical team, the surgical robot system is a complex platform and potentially captures large amounts of unique data from the surgical procedure that can be used to develop artificial intelligence solutions for the future surgical operating room benefiting from computer assisted interventions (CAI) [2]. Robotic systems in fact capture digital videos as well as instrument kinematic trajectories, instrument types and other system signals during surgical interventions, enabling more in-depth analysis of surgical motion and activity than with traditional instrumentation or video alone.…”

Section: Introductionmentioning

confidence: 99%

Gesture Recognition in Robotic Surgery With Multimodal Attention

Amsterdam

Funke

Edwards

et al. 2022

IEEE Trans. Med. Imaging

Self Cite

View full text Add to dashboard Cite

Automatically recognising surgical gestures from surgical data is an important building block of automated activity recognition and analytics, technical skill assessment, intra-operative assistance and eventually robotic automation. The complexity of articulated instrument trajectories and the inherent variability due to surgical style and patient anatomy make analysis and fine-grained segmentation of surgical motion patterns from robot kinematics alone very difficult. Surgical video provides crucial information from the surgical site with context for the kinematic data and the interaction between the instruments and tissue. Yet sensor fusion between the robot data and surgical video stream is non-trivial because the data have different frequency, dimensions and discriminative capability. In this paper, we integrate multimodal attention mechanisms in a two-stream temporal convolutional network to compute relevance scores and weight kinematic and visual feature representations dynamically in time, aiming to aid multimodal network training and achieve effective sensor fusion. We report the results of our system on the JIG-SAWS benchmark dataset and on a new in vivo dataset of suturing segments from robotic prostatectomy procedures. Our results are promising and obtain multimodal prediction sequences with higher accuracy and better temporal structure than corresponding unimodal solutions. Visualization of attention scores also gives physically interpretable insights on network understanding of strengths and weaknesses of each sensor.

show abstract

Computer Vision in the Surgical Operating Room

Cited by 40 publications

References 52 publications

Gesture Recognition in Robotic Surgery: A Review

Gesture Recognition in Robotic Surgery: A Review

MSDESIS: Multitask Stereo Disparity Estimation and Surgical Instrument Segmentation

Gesture Recognition in Robotic Surgery With Multimodal Attention

Contact Info

Product

Resources

About