CaRTS: Causality-Driven Robot Tool Segmentation from Vision and Kinematics Data

Ding, Hao; Jin-tan, Zhang; Kazanzides, Peter; Wu, Jie; Unberath, Mathias

doi:10.1007/978-3-031-16449-1_37

Cited by 13 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prevalent challenges in this setting have to do with simulation of deformable surfaces [124], with much work focused on developing intelligent control policies for robotic surgery. Surgeons and intelligent systems must also contend with smoke from energized devices [25,38] and blood. Finally, in order to facilitate machine learning, it is desirable for simulation frameworks to provide various ground truth data by default, including segmentation maps, depth maps, object poses, camera pose, and more [98].…”

Section: Visible Light Imagingmentioning

confidence: 99%

“…They collect real image and kinematic data using the dVRK and utilize the kinematic data to produce image data of virtual tools using a dVRK simulator [134]. Similarly, Ding et al [38] introduces a vision-and kinematics-based approach to robot tool segmentation based on a complementary causal model that preserves accuracy under domain shift to unseen domains. They collect a counterfactual dataset, where individual instances only differ by the presence or absence of a specific source of corruption, using a technique similar to that in [133], further demonstrating the utility of this type of data collection while reporting similar challenges, namely the lack of tool-to-tissue interaction through the re-play paradigm.…”

Section: Alternative Frameworkmentioning

confidence: 99%

“…Unfortunately, this is not the case in MIS, since the purpose of intra-operative imaging is to inform both spatial reasoning and clinical decision making [34,35]. Although specific spatial annotations, such as camera pose, tool-to-tissue relationships, and anatomical shape, may be obtained from real images through post hoc analysis [36], by incorporating robot kinematic models [37,38], or by leveraging the shape of known components [39,40], the assumptions made by these approaches do not extend beyond their specific use-case. The fact that intra-operative images are not routinely recorded nor stored further complicates the curation of large data sets that are needed for training machine learning algorithms, including deep convolutional neural networks, transformer architectures, which are the backbone of modern intelligent systems.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

et al. 2023

View full text Add to dashboard Cite

To mitigate the challenges of operating through narrow incisions under image guidance, there is a desire to develop intelligent systems that assist decision making and spatial reasoning in minimally invasive surgery (MIS). In this context, machine learning-based systems for interventional image analysis are receiving considerable attention because of their flexibility and the opportunity to provide immediate, informative feedback to clinicians. It is further believed that learning-based image analysis may eventually form the foundation for semior fully automated delivery of surgical treatments. A significant bottleneck in developing such systems is the availability of annotated images with sufficient variability to train generalizable models, particularly the most recently favored deep convolutional neural networks or transformer architectures. A popular alternative to acquiring and manually annotating data from the clinical practice is the simulation of these data from human-based models. Simulation has many advantages, including the avoidance of ethical issues, precisely controlled environments, and the scalability of data collection. Here, we survey recent work that relies on in silico training of learning-based MIS systems, in which data are generated via computational simulation. For each imaging modality, we review available simulation tools in terms of compute requirements, image quality, and usability, as well as their applications for training intelligent systems. We further discuss open challenges for simulation-based development of MIS systems, such as the need for integrated imaging and physical modeling for non-optical modalities, as well as generative patient models not dependent on underlying CT, MRI, or other patient data. In conclusion, as the capabilities of in silico training mature, with respect to sim-to-real transfer, computational efficiency, and degree of control, they are contributing toward the next generation of intelligent surgical systems.

show abstract

Section: Visible Light Imagingmentioning

confidence: 99%

Section: Alternative Frameworkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

et al. 2023

View full text Add to dashboard Cite

show abstract

“…On the robustness of medical procedures (Ding et al, 2022), built a causal tool segmentation model that iteratively aligns tool masks with observations. Unable to deal with occlusions and without leveraging temporal information the authors of this recent work also comment on the future next steps of robust causal machine learning tools.…”

Section: Out Of Distribution Robustness and Detectionmentioning

confidence: 99%

A Review of Causality for Learning Algorithms in Medical Image Analysis

Vlontzos

Rueckert

Kainz

2022

Melba

View full text Add to dashboard Cite

Medical image analysis is a vibrant research area that offers doctors and medical practitioners invaluable insight and the ability to accurately diagnose and monitor disease. Machine learning provides an additional boost for this area. However, machine learning for medical image analysis is particularly vulnerable to natural biases like domain shifts that affect algorithmic performance and robustness. In this paper we analyze machine learning for medical image analysis within the framework of Technology Readiness Levels and review how causal analysis methods can fill a gap when creating robust and adaptable medical image analysis algorithms.<br>We review methods using causality in medical imaging AI/ML and find that causal analysis has the potential to mitigate critical problems for clinical translation but that uptake and clinical downstream research has been limited so far.

show abstract

“…phase recognition [22,2,9], action detection [14,15], or tool detection [7,4], these approaches do not focus on holistic OR understanding. Most recently, Özsoy et al [16] proposed a new dataset, 4D-OR, and an approach for holistic OR modeling.…”

Section: Introductionmentioning

confidence: 99%

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

Özsoy¹,

Czempiel²,

Holm³

et al. 2023

Preprint

View full text Add to dashboard Cite

Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the operating room (OR) is, therefore, a challenging but essential task, with the potential to optimize the performance of surgical teams and aid in developing new surgical technologies to improve patient outcomes. The holistic representation of surgical scenes as semantic scene graphs (SGG), where entities are represented as nodes and relations between them as edges, is a promising direction for fine-grained semantic OR understanding. We propose, for the first time, the use of temporal information for more accurate and consistent holistic OR modeling. Specifically, we introduce memory scene graphs, where the scene graphs of previous time steps act as the temporal representation guiding the current prediction. We design an end-to-end architecture that intelligently fuses the temporal information of our lightweight memory scene graphs with the visual information from point clouds and images. We evaluate our method on the 4D-OR dataset and demonstrate that integrating temporality leads to more accurate and consistent results achieving an +5% increase and a new SOTA of 0.88 in macro F1. This work opens the path for representing the entire surgery history with memory scene graphs and improves the holistic understanding in the OR. Introducing scene graphs as memory representations can offer a valuable tool for many temporal understanding tasks.

show abstract

CaRTS: Causality-Driven Robot Tool Segmentation from Vision and Kinematics Data

Cited by 13 publications

References 36 publications

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

In silico simulation: a key enabling technology for next-generation intelligent surgical systems

A Review of Causality for Learning Algorithms in Medical Image Analysis

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

Contact Info

Product

Resources

About