Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

Islam, Mobarakol; Seenivasan, Lalithkumar; Ming, Lim Chwee; Ren, Hongliang

doi:10.1007/978-3-030-59716-0_60

Cited by 30 publications

(22 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deciding when the system should take a particular action is also vital for computer-assisted tools such as virtual reality or haptic guidance systems. Recently, Islam et al [104] proposed an enhanced graph neural network to perform spatial reasoning to infer the tooltissue interaction graph structure in a surgical scene. Based on the scene segmentation of the MICCAI Challenge 2018 dataset, they generated a graph-based tissue-tool interaction dataset with new annotations.…”

Section: A Development Of Autonomous Surgical Systemsmentioning

confidence: 99%

A Review on Deep Learning in Minimally Invasive Surgery

et al. 2021

View full text Add to dashboard Cite

In the last five years, deep learning has attracted great interest in computer-assisted systems for Minimally Invasive Surgery. The straightforward accessibility to images in surgical interventions makes deep neural networks enormously powerful for solving classification problems in complex surgical scenarios. The objective of this work is to provide readers a survey on deep learning models applied to minimally invasive surgery, identifying the different architectures used depending on the application, the results achieved until now, and the publicly available surgical datasets that can be used for validating new studies. A total of 85 publications have been extracted from manual research from four databases (IEEE Xplorer, Springer Link, Science Direct, and ACM Digital Library). After analyzing all these studies, they have been classified into four applications: surgical image analysis, surgical task analysis, surgical skill assessment, and automation of surgical tasks. This work provides a technical description of these works and a comparison among them. Finally, promising research directions to advance in this field are identified.

show abstract

Section: A Development Of Autonomous Surgical Systemsmentioning

confidence: 99%

A Review on Deep Learning in Minimally Invasive Surgery

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Dirichlet calibration, a multiclass calibration method, is introduced using Dirichlet distribution (Kull et al, 2019). Recently, label smoothing presents as one of the efficient regularization techniques to improve the confidence calibration and feature representation (Müller et al, 2019;Islam et al, 2020).…”

Section: A Related Workmentioning

confidence: 99%

Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition

Islam,

Seenivasan,

Ren

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Despite impressive accuracy, deep neural networks are often miscalibrated and tend to overly confident predictions. Recent techniques like temperature scaling (TS) and label smoothing (LS) show effectiveness in obtaining a well-calibrated model by smoothing logits and hard labels with scalar factors, respectively. However, the use of uniform TS or LS factor may not be optimal for calibrating models trained on a long-tailed dataset where the model produces overly confident probabilities for high-frequency classes. In this study, we propose class-distribution-aware TS (CDA-TS) and LS (CDA-LS) by incorporating class frequency information in model calibration in the context of long-tailed distribution. In CDA-TS, the scalar temperature value is replaced with the CDA temperature vector encoded with class frequency to compensate for the over-confidence. Similarly, CDA-LS uses a vector smoothing factor and flattens the hard labels according to their corresponding class distribution. We also integrate CDA optimal temperature vector with distillation loss, which reduces miscalibration in selfdistillation (SD). We empirically show that classdistribution-aware TS and LS can accommodate the imbalanced data distribution yielding superior performance in both calibration error and predictive accuracy. We also observe that SD with an extremely imbalanced dataset is less effective in terms of calibration performance. Code is available in https://github.com/mobarakol/Class-Distribution-Aware-TS-LS.

show abstract

“…If smoothened label, T LS = T (1 − ) + /K then CE loss with LS can be formulated as CE LS = − K k=1 T LS log(P ) where true label T , smoothing factor , total number of classes K and predicted probability P . A recent study [5] investigates that LS learns better feature representation in the penultimate layer, which is effective in object feature extraction. In this work, we observe the behavior of LS trained model in the DA task.…”

Section: Ls For Model Calibration and Damentioning

confidence: 99%

“…After the surgical instrument recognition problem has been solved to a large extent, the next goal is to get out the instrument itself and focus on the relationships between the instruments and the Region of Interest (ROI). Therefore, a graph-based network is introduced in [5] to do deep reasoning for the surgical scene, learn to infer a graph structure of instruments-tissue interaction, and predict the relationship between instruments and tissue. The further intention is that we want to express this prediction of their relationship in natural language, which can have richer scene information and the ability to interact with people.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery

Xu¹,

Islam²,

Lim³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Generating a surgical report in robot-assisted surgery, in the form of natural language expression of surgical scene understanding, can play a significant role in document entry tasks, surgical training, and post-operative analysis. Despite the state-of-the-art accuracy of the deep learning algorithm, the deployment performance often drops when applied to the Target Domain (TD) data. For this purpose, we develop a multi-layer transformer-based model with the gradient reversal adversarial learning to generate a caption for the multi-domain surgical images that can describe the semantic relationship between instruments and surgical Region of Interest (ROI). In the gradient reversal adversarial learning scheme, the gradient multiplies with a negative constant and updates adversarially in backward propagation, discriminating between the source and target domains and emerging domaininvariant features. We also investigate model calibration with label smoothing technique and the effect of a well-calibrated model for the penultimate layer's feature representation and Domain Adaptation (DA). We annotate two robotic surgery datasets of MICCAI robotic scene segmentation and Transoral Robotic Surgery (TORS) with the captions of procedures and empirically show that our proposed method improves the performance in both source and target domain surgical reports generation in the manners of unsupervised, zero-shot, one-shot, and few-shot learning.

show abstract

Learning and Reasoning with the Graph Structure Representation in Robotic Surgery

Cited by 30 publications

References 11 publications

A Review on Deep Learning in Minimally Invasive Surgery

A Review on Deep Learning in Minimally Invasive Surgery

Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition

Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery

Contact Info

Product

Resources

About