Artificial Intelligence for Intraoperative Guidance

Abstract

Objective: The aim of this study was to develop and evaluate the performance of artificial intelligence (AI) models that can identify safe and dangerous zones of dissection, and anatomical landmarks during laparoscopic cholecystectomy (LC). Summary Background Data: Many adverse events during surgery occur due to errors in visual perception and judgment leading to misinterpretation of anatomy. Deep learning, a subfield of AI, can potentially be used to provide real-time guidance intraoperatively. Methods: Deep learning models were developed and trained to identify safe (Go) and dangerous (No-Go) zones of dissection, liver, gallbladder, and hepatocystic triangle during LC. Annotations were performed by 4 high-volume surgeons. AI predictions were evaluated using 10-fold crossvalidation against annotations by expert surgeons. Primary outcomes were intersection-over-union (IOU) and F1 score (validated spatial correlation indices), and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, ± standard deviation. Results: AI models were trained on 2627 random frames from 290 LC videos, procured from 37 countries, 136 institutions, and 153 surgeons. Mean IOU, F1 score, accuracy, sensitivity, and specificity for the AI to identify Go zones were 0.53 (±0.24), 0.70 (±0.28), 0.94 (±0.05), 0.69 (±0.20). and 0.94 (±0.03), respectively. For No-Go zones, these metrics were 0.71 (±0.29), 0.83 (±0.31), 0.95 (±0.06), 0.80 (±0.21), and 0.98 (±0.05), respectively. Mean IOU for identification of the liver, gallbladder, and hepatocystic triangle were: 0.86 (±0.12), 0.72 (±0.19), and 0.65 (±0.22), respectively. Conclusions: AI can be used to identify anatomy within the surgical field. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

Using deep learning to identify the recurrent laryngeal nerve during thyroidectomy

“…However, leveraging these models’ success to soft tissue segmentation in digital open surgery images is not trivial, posing unique challenges due to the surgical setting and freehand digital images. While previous works in anatomical segmentation of surgical images have mainly examined laparoscopic and robot-assisted surgery 36 , 38 , 39 , these procedures generally have more constrained surgical scenes and standardized forms of image capture. Using a more accessible but freeform imaging modality in the unconstrained environment of open surgery brings lack of image structure and anatomical structure, lack of appearance consistency, and heightened difficulty of anatomical identification.…”

“…Further work has also introduced computer vision methods for use in laparoscopic and robot-assisted surgery, where surgical scenes are more complex and involve surgical activity and instruments constrained within a small field of view. These works span a wide range of tasks, including activity recognition 22 – 24 and phase detection 25 – 28 in surgical videos, surgical instrument detection 29 – 32 and segmentation 33 – 35 , as well as anatomical identification in a variety of procedures 36 , 37 such as laparoscopic cholecystectomy 38 – 40 . These works demonstrate the promise of using computer vision for surgical video analysis and intraoperative object and anatomy identification.…”

Challenges in surgical video annotation

“…For example, when annotating the cystic artery, the connective tissue surrounding the artery may make labeling the structure difficult as the border between the artery and the gallbladder may be 'fuzzy'. Approaches borrowed from surgical education, such as visual concordance testing, may help to better delineate these 'fuzzy' borders to arrive at a consensus annotation [16] .Clinical expert annotators may be able to better evaluate this border but there will likely be bias in how an image is labeled, particularly in datasets where videos are labeled by a small group of annotators.…”

“…Rather, such variability may reflect the 'fuzzy' nature of a phenomenon itself [41]. Even experienced surgeons may differ in their conceptualization of some phenomena, such as safe and unsafe zones of dissection or identification of specific anatomic structures [16,42,43]. Thus, combining annotations to serve as a fuzzier ground truth or to establish thresholds of agreement as ground truth may serve to either enhance modeling of clinical phenomena that are, by nature, fuzzy or provide a more realistic benchmark for model performance (i.e.…”

“…Thus, combining annotations to serve as a fuzzier ground truth or to establish thresholds of agreement as ground truth may serve to either enhance modeling of clinical phenomena that are, by nature, fuzzy or provide a more realistic benchmark for model performance (i.e. to compare agreement of a model to multiple annotators vs. a single annotator) [11,16,44].…”