Computer Vision in the Operating Room: Opportunities and Caveats

Kennedy-Metz, Lauren R.; Mascagni, Pietro; Torralba, Antonio; Dias, Roger D.; Perona, Pietro; Shah, Julie A.; Padoy, Nicolas; Zenati, Marco A.

doi:10.1109/tmrb.2020.3040002

Cited by 38 publications

(21 citation statements)

References 89 publications

(92 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To address the challenges associated with state and action observability, the development of an AI Coach would be greatly enhanced by nuanced surgical tool detection and people tracking methodology. The possibilities of incorporating these approaches into the OR are steadily increasing as surgical data science takes hold in surgical environments [14]. Additionally, robust and sensitive psychophysiological sensors equipped to team members may provide insight into impending mental model misalignments and function to provide automated feedback on team members' cognitive states.…”

Section: Discussionmentioning

confidence: 99%

“…In addition to the observable features, which can be measured using sensors and instrumentation in the operating room [13], [14], the team behavior depends on the proper understanding of the task. In particular, the team may have one of two mental states regarding protamine administration: bolus or incremental (especially because heparin is always given as a bolus).…”

Section: A Protamine Administrationmentioning

confidence: 99%

See 1 more Smart Citation

Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare

Seo,

Kennedy-Metz,

Zenati

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Shared mental models are critical to team success; however, in practice, team members may have misaligned models due to a variety of factors. In safety-critical domains (e.g., aviation, healthcare), lack of shared mental models can lead to preventable errors and harm. Towards the goal of mitigating such preventable errors, here, we present a Bayesian approach to infer misalignment in team members' mental models during complex healthcare task execution. As an exemplary application, we demonstrate our approach using two simulated team-based scenarios, derived from actual teamwork in cardiac surgery. In these simulated experiments, our approach inferred model misalignment with over 75% recall, thereby providing a building block for enabling computer-assisted interventions to augment human cognition in the operating room and improve teamwork.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: A Protamine Administrationmentioning

confidence: 99%

Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare

Seo,

Kennedy-Metz,

Zenati

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Various other applications of CV methods and AI can be seen in AIS, including application of CV and AI for education in surgery [ 74 ], to improve efficiency in the operating room [ 75 ], during neurosurgery [ 76 ], and in other surgical disciplines. For a comprehensive and recent review of the use of computer-vision-based methods in AIS and assisted surgery, the reader is referred to [ 77 ].…”

Section: Deep Learning and Computer Visionmentioning

confidence: 99%

Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?

Gumbs

Frigerio

Spolverato

et al. 2021

Sensors

View full text Add to dashboard Cite

Most surgeons are skeptical as to the feasibility of autonomous actions in surgery. Interestingly, many examples of autonomous actions already exist and have been around for years. Since the beginning of this millennium, the field of artificial intelligence (AI) has grown exponentially with the development of machine learning (ML), deep learning (DL), computer vision (CV) and natural language processing (NLP). All of these facets of AI will be fundamental to the development of more autonomous actions in surgery, unfortunately, only a limited number of surgeons have or seek expertise in this rapidly evolving field. As opposed to AI in medicine, AI surgery (AIS) involves autonomous movements. Fortuitously, as the field of robotics in surgery has improved, more surgeons are becoming interested in technology and the potential of autonomous actions in procedures such as interventional radiology, endoscopy and surgery. The lack of haptics, or the sensation of touch, has hindered the wider adoption of robotics by many surgeons; however, now that the true potential of robotics can be comprehended, the embracing of AI by the surgical community is more important than ever before. Although current complete surgical systems are mainly only examples of tele-manipulation, for surgeons to get to more autonomously functioning robots, haptics is perhaps not the most important aspect. If the goal is for robots to ultimately become more and more independent, perhaps research should not focus on the concept of haptics as it is perceived by humans, and the focus should be on haptics as it is perceived by robots/computers. This article will discuss aspects of ML, DL, CV and NLP as they pertain to the modern practice of surgery, with a focus on current AI issues and advances that will enable us to get to more autonomous actions in surgery. Ultimately, there may be a paradigm shift that needs to occur in the surgical community as more surgeons with expertise in AI may be needed to fully unlock the potential of AIS in a safe, efficacious and timely manner.

show abstract

“…Modeling surgical procedures with all their different components is crucial to develop intelligent systems that understand surgical operations, enable computer interventions, and provide situational aware assistance [1]. Yet, as surgical procedures take place in highly complex, dynamic, cluttered and occluded environments, with the broad heterogenity of procedures, patients, medical teams, imaging and life support devices, surgical tools, operating table and monitors, they become a challenging input for visual perception and understanding [2,3]. Even though state-of-the-art computer vision techniques have proved to be highly accurate for task-specifically designed datasets, it is non-trivial to translate these methods to the highly complex real OR environments.…”

Section: Introductionmentioning

confidence: 99%

Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures

Özsoy,

Örnek,

Eck

et al. 2021

Preprint

View full text Add to dashboard Cite

From a computer science viewpoint, a surgical domain model needs to be a conceptual one incorporating both behavior and data. It should therefore model actors, devices, tools, their complex interactions and data flow. To capture and model these, we take advantage of the latest computer vision methodologies for generating 3D scene graphs from camera views. We then introduce the Multimodal Semantic Scene Graph (MSSG) which aims at providing a unified symbolic, spatiotemporal and semantic representation of surgical procedures. This methodology aims at modeling the relationship between different components in surgical domain including medical staff, imaging systems, and surgical devices, opening the path towards holistic understanding and modeling of surgical procedures. We then use MSSG to introduce a dynamically generated graphical user interface tool for surgical procedure analysis which could be used for many applications including process optimization, OR design and automatic report generation. We finally demonstrate that the proposed MSSGs could also be used for synchronizing different complex surgical procedures. While the system still needs to be integrated into real operating rooms before getting validated, this conference paper aims mainly at providing the community with the basic principles of this novel concept through a first prototypal partial realization based on MVOR dataset.

show abstract

Computer Vision in the Operating Room: Opportunities and Caveats

Cited by 38 publications

References 89 publications

Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare

Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare

Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?

Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures

Contact Info

Product

Resources

About