4D-OR: Semantic Scene Graphs for OR Domain Modeling

Özsoy, Ege; Örnek, Evin Pınar; Eck, Ulrich; Czempiel, Tobias; Tombari, Federico; Navab, Nassir

doi:10.1007/978-3-031-16449-1_45

Cited by 14 publications

(10 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dataset: We use the 4D-OR [16] dataset following the official train, validation, and test splits. It comprises ten simulated knee surgeries recorded using six Table 1.…”

Section: Methodsmentioning

confidence: 99%

“…We build up on the 4D-OR [16] method, which uses a single timepoint for generating semantic scene graphs. The 4D-OR method extracts human and object poses and uses them to compute point cloud features for all object pairs.…”

Section: Single Timepoint Scene Graph Generationmentioning

confidence: 99%

“…Effective operating room (OR) management is critical for improving patient outcomes, optimizing surgical team performance, and developing new surgical technologies [10]. Scene understanding, particularly in dynamic OR environments, is a challenging task that requires holistic and semantic modeling [11,16], where both the coarse and the fine-level activities and interactions are understood. While many recent works addressed different aspects of this understanding, such as surgical Fig.…”

Section: Introductionmentioning

confidence: 99%

“…phase recognition [22,2,9], action detection [14,15], or tool detection [7,4], these approaches do not focus on holistic OR understanding. Most recently, Özsoy et al [16] proposed a new dataset, 4D-OR, and an approach for holistic OR modeling. They model the OR using semantic scene graphs, which summarize the entire scene at each timepoint, connecting the different entities with their semantic relationships.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

Özsoy¹,

Czempiel²,

Holm³

et al. 2023

Preprint

View full text Add to dashboard Cite

Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the operating room (OR) is, therefore, a challenging but essential task, with the potential to optimize the performance of surgical teams and aid in developing new surgical technologies to improve patient outcomes. The holistic representation of surgical scenes as semantic scene graphs (SGG), where entities are represented as nodes and relations between them as edges, is a promising direction for fine-grained semantic OR understanding. We propose, for the first time, the use of temporal information for more accurate and consistent holistic OR modeling. Specifically, we introduce memory scene graphs, where the scene graphs of previous time steps act as the temporal representation guiding the current prediction. We design an end-to-end architecture that intelligently fuses the temporal information of our lightweight memory scene graphs with the visual information from point clouds and images. We evaluate our method on the 4D-OR dataset and demonstrate that integrating temporality leads to more accurate and consistent results achieving an +5% increase and a new SOTA of 0.88 in macro F1. This work opens the path for representing the entire surgery history with memory scene graphs and improves the holistic understanding in the OR. Introducing scene graphs as memory representations can offer a valuable tool for many temporal understanding tasks.

show abstract

“…Dataset: We use the 4D-OR [16] dataset following the official train, validation, and test splits. It comprises ten simulated knee surgeries recorded using six Table 1.…”

Section: Methodsmentioning

confidence: 99%

Section: Single Timepoint Scene Graph Generationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

Özsoy¹,

Czempiel²,

Holm³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…This could allow AR systems to have higher levels of perceptual understanding of the environment they act in. In some of the most recent work in surgical scene understanding, Ozsoy et al [ 11 , 12 ] used multiple camera views of surgical scenes to not only build a 4D reconstruction of the scene but also generate semantic scene graphs recovering, modeling and representing complex interaction between surgical staff, patient, device, and tools within the OR. Such high-level computer perception will allow for intelligent workflow-driven solutions [ 13 ].…”

Section: The Medical Augmented Reality Frameworkmentioning

confidence: 99%

Medical Augmented Reality: Definition, Principle Components, Domain Modeling, and Design-Development-Validation Process

et al. 2022

Self Cite

View full text Add to dashboard Cite

Three decades after the first set of work on Medical Augmented Reality (MAR) was presented to the international community, and ten years after the deployment of the first MAR solutions into operating rooms, its exact definition, basic components, systematic design, and validation still lack a detailed discussion. This paper defines the basic components of any Augmented Reality (AR) solution and extends them to exemplary Medical Augmented Reality Systems (MARS). We use some of the original MARS applications developed at the Chair for Computer Aided Medical Procedures and deployed into medical schools for teaching anatomy and into operating rooms for telemedicine and surgical guidance throughout the last decades to identify the corresponding basic components. In this regard, the paper is not discussing all past or existing solutions but only aims at defining the principle components and discussing the particular domain modeling for MAR and its design-development-validation process, and providing exemplary cases through the past in-house developments of such solutions.

show abstract

A Survey on 3D Scene Graphs: Definition, Generation and Application

Bae

Shin

et al. 2023

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

4D-OR: Semantic Scene Graphs for OR Domain Modeling

Cited by 14 publications

References 23 publications

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms

Medical Augmented Reality: Definition, Principle Components, Domain Modeling, and Design-Development-Validation Process

A Survey on 3D Scene Graphs: Definition, Generation and Application

Contact Info

Product

Resources

About