Learning to See Forces: Surgical Force Prediction with RGB-Point Cloud Temporal Convolutional Networks

Gao, Cong; Liu, Xingtong; Peven, Michael; Unberath, Mathias; Reiter, Austin

doi:10.1007/978-3-030-01201-4_14

Cited by 21 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The research on the training of CNNs for the image-based force estimation with more enriched input data is still in progress. In 2018, Gao et al [64] proposed a vision-based surgical force prediction model, called RGB-Point Cloud Temporal Convolutional Network (RPC-TCN), which combined RGB-D information with time-series analysis (see Fig. 15).…”

Section: Evolution Of Learning-based Methodsmentioning

confidence: 99%

Image-Based Force Estimation in Medical Applications: A Review

Nazari¹,

Janabi–Sharifi²,

Zareinia³

2023

Preprint

View full text Add to dashboard Cite

<p>Minimally invasive robotic interventions have highlighted the need to develop efficient techniques to measure forces applied to the soft tissues. Since the last decade, many scholars have focused on micro-scale and macro-scale robotic manipulations. Early articles used the model of soft tissue mathematically and tracked the displacement of the contour of the object in the vision system to provide the corresponding force to the user. Lack of knowledge of different materials and the computational complexity led to a transition from model-based to learning-based approaches to interpret the relation between object deformations, extracted from the vision system, and the real forces applied to the object. The dramatic growth of machine learning techniques and its integration with computer vision has brought novel learning-based visual data processing methods to the area. The application of the image-based force estimation methods in a controlled medical intervention has also received significant attention in the last five years. A decent number of surveys have been published on micromanipulation in recent years, especially for cell microinjection. However, the state of the art in meso- and macro-scale medical robotic interventions has not been reviewed. The aim and contribution of this paper are to fill the stated gap by reviewing the recent advances in image-based force estimation in robotic interventions. The survey shows that learning-based force estimation methods are growing significantly by using deep learning-based methods. The survey will encourage researchers and surgeons to apply learning-based algorithms to real-time medical and health-related operations.</p>

show abstract

Section: Evolution Of Learning-based Methodsmentioning

confidence: 99%

Image-Based Force Estimation in Medical Applications: A Review

Nazari¹,

Janabi–Sharifi²,

Zareinia³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In [102], the authors proposed a force estimation based on visual cues to infer tissue deformation, using a Temporal Convolutional Network (TCN). The input of the network are RGB and depth images collected using a Kinetic2 camera.…”

Section: Automation Of Surgical Tasksmentioning

confidence: 99%

A Review on Deep Learning in Minimally Invasive Surgery

et al. 2021

View full text Add to dashboard Cite

In the last five years, deep learning has attracted great interest in computer-assisted systems for Minimally Invasive Surgery. The straightforward accessibility to images in surgical interventions makes deep neural networks enormously powerful for solving classification problems in complex surgical scenarios. The objective of this work is to provide readers a survey on deep learning models applied to minimally invasive surgery, identifying the different architectures used depending on the application, the results achieved until now, and the publicly available surgical datasets that can be used for validating new studies. A total of 85 publications have been extracted from manual research from four databases (IEEE Xplorer, Springer Link, Science Direct, and ACM Digital Library). After analyzing all these studies, they have been classified into four applications: surgical image analysis, surgical task analysis, surgical skill assessment, and automation of surgical tasks. This work provides a technical description of these works and a comparison among them. Finally, promising research directions to advance in this field are identified.

show abstract

“…This technique involves several different types of image processing methods, such as feature extraction, filtering of light reflections, and also can be approached using neural networks, but the crucial parts are usually the reconstruction of the tissue surface and handling the inhomogenity in the tissue. Based on the detected deformations of the targeted tissue surface, the applied force values can be estimated [33], [61]- [63], [65]. Despite the mentioned benefits of this technique, the implementation is extremely complex, and usually computationally intense.…”

Section: Palpation With Vision-based Force Estimationmentioning

confidence: 99%

Recent Advances in Robot-Assisted Surgery: Soft Tissue Contact Identification

Nagy

Haidegger

2019

2019 IEEE 13th International Symposium on Applied Computational Intelligence and Informatics (SACI)

View full text Add to dashboard Cite

Robot-Assisted MinimallyInvasive Surgery (RAMIS) is becoming standard-of-care in western medicine. RAMIS offers better patient outcome compared to traditional open surgery, however, the surgeons' ability to identify the tissues with the sense of touch is missing from most robotic systems. Regarding haptic feedback, the most promising diagnostic technique is probably palpation; a physical contact examination method through which information can be gathered about the underlying structures by gently pressing with the fingers. In open surgery, palpation is widely used to identify blood vessels, tendons or even tumors; and the knowledge on the exact location of such elements is often crucial with respect to the outcome of the intervention. This paper presents a review of the actual research directions in the field of palpation in RAMIS.

show abstract

Learning to See Forces: Surgical Force Prediction with RGB-Point Cloud Temporal Convolutional Networks

Cited by 21 publications

References 13 publications

Image-Based Force Estimation in Medical Applications: A Review

Image-Based Force Estimation in Medical Applications: A Review

A Review on Deep Learning in Minimally Invasive Surgery

Recent Advances in Robot-Assisted Surgery: Soft Tissue Contact Identification

Contact Info

Product

Resources

About