Validation of Machine Learning–Based Automated Surgical Instrument Annotation Using Publicly Available Intraoperative Video

Abstract

BACKGROUND: Intraoperative tool movement data have been demonstrated to be clinically useful in quantifying surgical performance. However, collecting this information from intraoperative video requires laborious hand annotation. The ability to automatically annotate tools in surgical video would advance surgical data science by eliminating a time-intensive step in research. OBJECTIVE: To identify whether machine learning (ML) can automatically identify surgical instruments contained within neurosurgical video. METHODS: A ML model which automatically identifies surgical instruments in frame was developed and trained on multiple publicly available surgical video data sets with instrument location annotations. A total of 39 693 frames from 4 data sets were used (endoscopic endonasal surgery [EEA] [30 015 frames], cataract surgery [4670], laparoscopic cholecystectomy [2532], and microscope-assisted brain/spine tumor removal [2476]). A second model trained only on EEA video was also developed. Intraoperative EEA videos from YouTube were used for test data (3 videos, 1239 frames). RESULTS: The YouTube data set contained 2169 total instruments. Mean average precision (mAP) for instrument detection on the YouTube data set was 0.74. The mAP for each individual video was 0.65, 0.74, and 0.89. The second model trained only on EEA video also had an overall mAP of 0.74 (0.62, 0.84, and 0.88 for individual videos). Development costs were $130 for manual video annotation and under $100 for computation. CONCLUSION: Surgical instruments contained within endoscopic endonasal intraoperative video can be detected using a fully automated ML model. The addition of disparate surgical data sets did not improve model performance, although these data sets may improve generalizability of the model in other use cases.

Intraoperative video recording in otolaryngology for surgical education: Evolution and considerations

“…Intraoperative tool movement tracking data has been shown to be clinically useful in quantifying surgical performance. A study demonstrated that machine learning can be utilized to identify surgical instruments within endoscopic endonasal intraoperative video and increase access to this information of surgical performance [22]. Deep neural networks have also been used in analyzing the operative steps in laparoscopic sleeve gastrectomy, with a 85.6% accuracy validated against surgeon annotations of the videos [23].…”

In Reply: Deep Neural Networks Can Accurately Detect Blood Loss and Hemorrhage Control Task Success from Video

“…[6][7][8] Within Neurosurgery, our group demonstrated that neural networks could reliably detect surgical instruments in view in a cadaveric training exercise and that these results translated to real intraoperative video. 9,10 The use of time series deep learning approaches extends to efforts outside of the analysis of surgical video, such as measuring intracranial pressure. 11,12 We wholeheartedly concur with Dr Lim's view that, despite the widespread availability of modeling approaches, exploiting intraoperative time series data remains an undiscovered scientific frontier with deep clinical applications.…”

“…6-8 Within Neurosurgery, our group demonstrated that neural networks could reliably detect surgical instruments in view in a cadaveric training exercise and that these results translated to real intraoperative video. 9,10 The use of time series deep learning approaches extends to efforts outside of the analysis of surgical video, such as measuring intracranial pressure. 11,12…”

Commentary: Validation of Machine Learning-Based Automated Surgical Instrument Annotation Using Publicly Available Intraoperative Video

“…7-9 Subsequently, 3 intraoperative endoscopic endonasal videos from YouTube (YouTube; Google LLC) were used to test the detector and demonstrate its generalizability. Markarian et al 6 concluded that surgical instruments contained within the endoscopic endonasal intraoperative video were able to be detected using a fully automated ML model. They also noted that although multiple data sets from disparate contexts were not beneficial in improving the model and its tool detection, it may potentially improve the generalizability of the model for alternative use.…”

“…Furthermore, for many neurosurgeons, aiming to achieve technical excellence can be a significant contributor to their potential success. Models, such as the one produced by Markarian et al, 6 can provide objective and quantifiable measurements to assist in not only differentiating between experienced and inexperienced surgeons but also help residents and early surgeons improve their technical expertise. For machine learning models, operative videos can be a rich source of data for analyzing surgical performance and understanding technical skills.…”