Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery

Wu, Jie; Tamhane, Aniruddha; Kazanzides, Peter; Unberath, Mathias

doi:10.1007/s11548-021-02343-y

Cited by 15 publications

(11 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…to general motion compensation for working in regions undergoing physiological motion [68] or control considering uncertainties [204]. [13]- [18] [13]- [15] [13]- [15], [17], [18] [13]- [15], [17], [18] [19] [19], [20] [21]…”

Section: Database Search and Filteringmentioning

confidence: 99%

“…As a fundamental component of training, skill assessment has received attention, especially in automation through data analysis. Some studies focused on proficiency analysis [19], [147], [156], [223], [231], [259], [260] as well as addressed the mental and physical workload of the user [135], [262] or the influence of training on haptic perception [62].…”

Section: Skill Assessmentmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

D'Ettorre

Mariani

Stilli

et al. 2021

IEEE Robot. Automat. Mag.

Self Cite

View full text Add to dashboard Cite

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.R obotic-assisted surgery is now well established in clinical practice and has become the gold-standard clinical treatment option for several clinical indications. The field of robotic-assisted surgery is expected to grow substantially in the next decade, with a range of new robotic devices emerging to address unmet clinical needs across different specialties. A vibrant surgical robotics research community is pivotal for conceptualizing such new systems as well as for developing and training the engineers and scientists to translate them into practice. The da Vinci Research Kit (dVRK), an academic and industry collaborative effort to repurpose decommissioned da Vinci surgical systems [Intuitive Surgical Inc. (ISI), California, USA] as a research platform for surgical robotics research, has been a key initiative for addressing a barrier to entry for new research groups in surgical robotics. In this article, we present an extensive review of the publications that have been facilitated by the dVRK over the past decade. We classify research efforts into different categories and outline some of the major challenges and needs for the robotics community to maintain and build upon this initiative.

show abstract

Section: Database Search and Filteringmentioning

confidence: 99%

Section: Skill Assessmentmentioning

confidence: 99%

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

D'Ettorre

Mariani

Stilli

et al. 2021

IEEE Robot. Automat. Mag.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Reconstructions of newly created bone fragments should be possible through similarity modeling approaches, such as by extending the framework introduced by Gao et al (2020c) to include multiple objects and non-rigid deformation. Finally, recurrent NN approaches, such as long short-term memory (LSTM) components, could provide some temporal segmentation of the intervention into phases and gestures, as is already done for laparoscopic surgery ( Vercauteren et al, 2019 ; Garrow et al, 2021 ; Wu et al, 2021 ). This segmentation could be useful in identifying when 1) certain objects need to be tracked, 2) a radically different view has been collected, or 3) new objects have been split off from an existing object.…”

Section: Perspectivementioning

confidence: 99%

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Unberath

Gao

et al. 2021

Front. Robot. AI

Self Cite

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

“…Reconstructions of newly created bone fragments should be possible through similarity modeling approaches, such as by extending the framework introduced by [65] to include multiple objects and non-rigid deformation. Finally, recurrent NN approaches, such as long short-term memory (LSTM) components, could provide some temporal segmentation of the intervention into phases and gestures, as is already done for laparoscopic surgery [14,106,107]. This segmentation could be useful in identifying when 1) certain objects need to be tracked, 2) a radically different view has been collected, or 3) new objects have been split off from an existing object.…”

Section: Registration Of Multiple Objects Compound or Non-rigid Motio...mentioning

confidence: 99%

The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective

Unberath¹,

Cong²,

Yi-cheng³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e. g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

show abstract

Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery

Cited by 15 publications

References 19 publications

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

The Impact of Machine Learning on 2D/3D Registration for Image-guided Interventions: A Systematic Review and Perspective

Contact Info

Product

Resources

About