Automated operative phase identification in peroral endoscopic myotomy

Abstract

Background Artificial intelligence (AI) and computer vision (CV) have revolutionized image analysis. In surgery, CV applications have focused on surgical phase identification in laparoscopic videos. We proposed to apply CV techniques to identify phases in an endoscopic procedure, peroral endoscopic myotomy (POEM). Methods POEM videos were collected from Massachusetts General and Showa University Koto Toyosu Hospitals. Videos were labeled by surgeons with the following ground truth phases: 1) Submucosal injection, 2) Mucosotomy, 3) Submucosal tunnel, 4) Myotomy, and 5) Mucosotomy closure. The deep-learning CV model -Convolutional Neural Network (CNN) plus Long Short-Term Memory (LSTM) -was trained on 30 videos to create POEMNet. We then used POEMNet to identify operative phases in the remaining 20 videos. The model's performance was compared to surgeon annotated ground truth. Results POEMNet's overall phase identification accuracy was 87.6% (95% CI 87.4% to 87.9%). When evaluated on a per-phase basis, the model performed well, with mean unweighted and prevalence-weighted F1 scores of 0.766 and 0.875, respectively. The model performed best with longer phases, with 70.6% accuracy for phases that had a duration under five minutes and 88.3% accuracy for longer phases. Discussion A deep-learning based approach to CV, previously successful in laparoscopic video phase identification, translates well to endoscopic procedures. With continued refinements, AI could contribute to intra-operative decision-support systems and postoperative risk prediction.

Surgical data science and artificial intelligence for surgical education

“…The resulting model, EndoNet, demonstrated 92% accuracy in the classification of surgical phases when used for posthoc analysis, and 86% accuracy when used for real‐time inference 44 . Deep learning models have since been trained to accurately recognize phases across a range of procedures, including bariatric, 45 colorectal, 46 ophthalmic, 47 and intraluminal interventions 48 …”

“…The Surgical AI and Innovation Laboratory at Massachusetts General Hospital have described the concept of the “surgical fingerprint” wherein phase recognition algorithms can assess the video of interest's workflow against that of a pre‐existing database to determine whether the video is following an expected operative course. Time points with deviations from an expected operative course could signal areas where complications or unexpected events might occur 45,48 …”

Challenges in surgical video annotation

“…However, this does not account for potential agreement that can occur by chance alone. Therefore, various statistical measures of agreement can be considered, such as Cohen or Fleiss's j, Krippendorf's a, or intraclass correlation coefficient [11,12]. An in-depth discussion of the appropriateness of individual metrics for a given situation is outside the scope of this article; however, it is important to consider aspects such as the number of annotators and the prevalence of a given annotation [13,14].…”

A Guide to Annotation of Neurosurgical Intraoperative Video for Machine Learning Analysis and Computer Vision

“…The first step in developing a CV pipeline is to create a library (dataset) composed of individual images (frames) from surgeries, to which "annotations" are applied (termed "ground-truth" data). These annotations are overlays that outline selected tools anatomical structures, or stages of an operation on a frame-by-frame basis 2,13,14 . Ground-truth data are often generated manually, and sometimes requires expert assessment.…”