Speech-Based Surgical Phase Recognition for Non-Intrusive Surgical Skills’ Assessment in Educational Contexts

Abstract

Surgeons’ procedural skills and intraoperative decision making are key elements of clinical practice. However, the objective assessment of these skills remains a challenge to this day. Surgical workflow analysis (SWA) is emerging as a powerful tool to solve this issue in surgical educational environments in real time. Typically, SWA makes use of video signals to automatically identify the surgical phase. We hypothesize that the analysis of surgeons’ speech using natural language processing (NLP) can provide deeper insight into the surgical decision-making processes. As a preliminary step, this study proposes to use audio signals registered in the educational operating room (OR) to classify the phases of a laparoscopic cholecystectomy (LC). To do this, we firstly created a database with the transcriptions of audio recorded in surgical educational environments and their corresponding phase. Secondly, we compared the performance of four feature extraction techniques and four machine learning models to find the most appropriate model for phase recognition. The best resulting model was a support vector machine (SVM) coupled to a hidden-Markov model (HMM), trained with features obtained with Word2Vec (82.95% average accuracy). The analysis of this model’s confusion matrix shows that some phrases are misplaced due to the similarity in the words used. The study of the model’s temporal component suggests that further attention should be paid to accurately detect surgeons’ normal conversation. This study proves that speech-based classification of LC phases can be effectively achieved. This lays the foundation for the use of audio signals for SWA, to create a framework of LC to be used in surgical training, especially for the training and assessment of procedural and decision-making skills (e.g., to assess residents’ procedural knowledge and their ability to react to adverse situations).

Deep Learning in Surgical Workflow Analysis: A Review of Phase and Step Recognition

“…In our review, almost all of the selected studies preferred CNN based architectures for feature extraction. Only Guzmán-García et al [46] used linear Word2Vec [65] model for creating a feature space from the transcription of speech uttered during surgery.…”

“…In addition, Guzmán-García et al [46] extracted word vectors and compared HMM together with SVM, random forest, logistic regression, and shallow linear layers for classification. Paysan et al [54] used hidden semi-Markov model for modeling temporal relations.…”

“…The most frequently used strategy in surgical workflow analysis is supervised learning (i.e. [24], [27]- [29], [31], [33], [34], [36]- [39], [41], [42], [44]- [46], [50], [74]). With supervised learning, an input is mapped to a predefined class, i. e. surgical phase one, two, etc.…”

SENECA: An Attention Support Tool for Context-related Content Learning

“…NLP techniques have been widely used in intelligent tutoring systems that helped acquire content knowledge [9]. For example, in Guzmán-García et al [22], the analysis of the speaking of surgeons into the operating room through NLP techniques is proposed to obtain a deeper vision of intraoperative decision-making processes. The goal of this study was to create a technique for identifying and analyzing the various surgical phases, as well as a workflow that was equivalent to the procedure's framework, in order to improve surgical learning in The Educational Operating Room.…”

SENECA: An Attention Support Tool for Context-related Content Learning

“…NLP techniques have been widely used in intelligent tutoring systems that helped acquire content knowledge [8]. For example, in Guzmán-García et al [21], the analysis of the surgeons' speaking taken into the operating room through NLP techniques is proposed to obtain a deeper vision of intraoperative decision-making processes. This study aimed to develop a method of recognizing and evalu- ating the various surgical phases and developing a workflow comparable with the framework of the procedure to improve surgical learning in The Educational Operating Room.…”