Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition

Demir, Kubilay Can; Schieber, Hannah; Weise, Tobias; Roth, Daniel; May, Matthias; Maier, Andreas; Yang, Seung Hee

doi:10.1109/jbhi.2023.3311628

Cited by 6 publications

(1 citation statement)

References 109 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Technologies that enable next-generation context-aware systems in the operating room are currently intensively researched in the domain of surgical workflow recognition [ 1 ]. Recent studies that apply machine learning algorithms to this task have shown highly promising results [ 2 , 3 ]. To further support advances in this area, academic machine learning competitions are hosted regularly [ 4 – 6 ].…”

Section: Introductionmentioning

confidence: 99%

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Kostiuchik,

Sharan,

Mayer

et al. 2024

Int J CARS

View full text Add to dashboard Cite

Purpose Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split. Methods We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits. Results We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks. Conclusion In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/.

show abstract

Section: Introductionmentioning

confidence: 99%

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Kostiuchik,

Sharan,

Mayer

et al. 2024

Int J CARS

View full text Add to dashboard Cite

show abstract

OSFENet: Object Spatiotemporal Feature Enhanced Network for Surgical Phase Recognition

You,

Zhang,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Audio-based event detection in the operating room

Fuchtmann,

Riedel,

Berlet

et al. 2024

Int J CARS

View full text Add to dashboard Cite

Purpose Even though workflow analysis in the operating room has come a long way, current systems are still limited to research. In the quest for a robust, universal setup, hardly any attention has been given to the dimension of audio despite its numerous advantages, such as low costs, location, and sight independence, or little required processing power. Methodology We present an approach for audio-based event detection that solely relies on two microphones capturing the sound in the operating room. Therefore, a new data set was created with over 63 h of audio recorded and annotated at the University Hospital rechts der Isar. Sound files were labeled, preprocessed, augmented, and subsequently converted to log-mel-spectrograms that served as a visual input for an event classification using pretrained convolutional neural networks. Results Comparing multiple architectures, we were able to show that even lightweight models, such as MobileNet, can already provide promising results. Data augmentation additionally improved the classification of 11 defined classes, including inter alia different types of coagulation, operating table movements as well as an idle class. With the newly created audio data set, an overall accuracy of 90%, a precision of 91% and a F1-score of 91% were achieved, demonstrating the feasibility of an audio-based event recognition in the operating room. Conclusion With this first proof of concept, we demonstrated that audio events can serve as a meaningful source of information that goes beyond spoken language and can easily be integrated into future workflow recognition pipelines using computational inexpensive architectures.

show abstract

Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition

Cited by 6 publications

References 109 publications

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Surgical phase and instrument recognition: how to identify appropriate dataset splits

OSFENet: Object Spatiotemporal Feature Enhanced Network for Surgical Phase Recognition

Audio-based event detection in the operating room

Contact Info

Product

Resources

About