CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions

Abstract

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.Index Terms-Artificial intelligence, computer assisted interventions, interventional workflow, intra-operative imaging, surgical planning, data fusion, surgical scene understanding, contextaware user interface, machine and deep learning, surgical data science

Computer Vision in the Surgical Operating Room

“…The majority of surgical AR systems have been founded on geometric vision algorithms but deep learning methods are emerging, e.g., for US to CT in spine surgery [52] or to design efficient deformable registration in laparoscopic liver surgery [53]. Despite methodological advances, significant open problems persist in surgical AR, such as adding contextual information to the visualization (e.g., identifying anatomical structures and critical surgical areas and detecting surgical phases and complications) [54], ensuring robust localization despite occlusions and DOI: 10.1159/000511934 displaying relevant information to different stakeholders in the OR. Work is advancing to address these challenges and evaluation of the state-of-the-art learning-based method for visual human pose estimation in the OR has recently been reported [55] alongside a review dedicated to face detection into the OR [56] and methods to estimate both surgical phases and remaining surgery durations [57] which can be used to alter information displayed at different times.…”

HoloYolo: A proof‐of‐concept study for marker‐less surgical navigation of spinal rod implants with augmented reality and on‐device machine learning

“…In recent years, augmented reality has become a mature technology offering new and exciting possibilities in the field of surgical navigation 10,11,36,37 . With the rise of machine learning, surgical navigation is shifting from traditional methods to intelligent approaches that consider the surgical context and develop an understanding of surgical procedures 38 . In this work, we employed machine learning to transform the simple but tedious task of manually acquiring screw head positions with a pointing device into a completely machine‐driven procedure.…”

Artificial Intelligence-Assisted Surgery: Potential and Challenges

“…In their entirety, these heterogeneous sensors provide the information necessary to infer the actual course of the operation and to provide proper assistance at the right time. This is subsumed under the term "context-aware assistance" [7], which avoids an information overflow and decreases the cognitive load, in particular in an already stressful and complex environment such as the OR. A prerequisite for providing such assistance is a SensorOR in which all devices are connected to collect their data ( Fig.…”

“…In other AI domains [19], there are many open data sets that can be used to develop, evaluate and compare different machine learning algorithms. While access to data sources is crucial also in surgery, only few public annotated data sets for different applications such as surgical phase detection [20, 21], surgical training [22, 23], and segmentation [21, 24] exist [7]. The Endoscopic Vision Challenge [25], an initiative that supports the availability of new public data sets for the systematic comparison of algorithms, has hosted challenges in surgical vision.…”