Machine Learning for Surgical Phase Recognition

Abstract

Objective: To provide an overview of ML models and data streams utilized for automated surgical phase recognition. Background: Phase recognition identifies different steps and phases of an operation. ML is an evolving technology that allows analysis and interpretation of huge data sets. Automation of phase recognition based on data inputs is essential for optimization of workflow, surgical training, intraoperative assistance, patient safety, and efficiency. Methods: A systematic review was performed according to the Cochrane recommendations and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. PubMed, Web of Science, IEEExplore, GoogleScholar, and CiteSeerX were searched. Literature describing phase recognition based on ML models and the capture of intraoperative signals during general surgery procedures was included. Results: A total of 2254 titles/abstracts were screened, and 35 full-texts were included. Most commonly used ML models were Hidden Markov Models and Artificial Neural Networks with a trend towards higher complexity over time. Most frequently used data types were feature learning from surgical videos and manual annotation of instrument use. Laparoscopic cholecystectomy was used most commonly, often achieving accuracy rates over 90%, though there was no consistent standardization of defined phases. Conclusions: ML for surgical phase recognition can be performed with high accuracy, depending on the model, data type, and complexity of surgery. Different intraoperative data inputs such as video and instrument type can successfully be used. Most ML models still require significant amounts of manual expert annotations for training. The ML models may drive surgical workflow towards standardization, efficiency, and objectiveness to improve patient outcome in the future. Registration PROSPERO: CRD42018108907

Robotic-assisted cholecystectomy is superior to laparoscopic cholecystectomy in the initial training for surgical novices in an ex vivo porcine model: a randomized crossover study

“…There is, however, high potential for use of force feedback that can be derived from this system as seen by the complications that happened during the training procedures in the present study. This will be evaluated as a feedback mechanism to enhance training in future studies as well as automated methods of skill assessment to facilitate performance assessment and feedback [ 76 , 77 ].…”

Surgical data science and artificial intelligence for surgical education

“…Starting from fairly standardized procedures such as cholecystectomy and hysterectomy, early algorithms made use of hand‐crafted signals such as surgical instrument usage and time dependencies to model and visualize surgical workflow using classical machine learning techniques such as dynamic time warping and hidden Markov models 41,42 . More recently, breakthroughs in deep neural networks have boosted computer vision performance and revived the field of surgical workflow analysis 43 …”

PhacoTrainer: A Multicenter Study of Deep Learning for Activity Recognition in Cataract Surgical Videos

“…Our model directly learned useful features from images during the training process, which has become one of the most common data types used in surgical phase recognition in recent years. 22 This avoids the need to extract other information, such as instrument use, color histograms, texture, etc. before running the model and allows faster and easier model deployment.…”