Developing Surgical Skill Level Classification Model Using Visual Metrics and a Gradient Boosting Algorithm

Shafiei, Somayeh B.; Shadpour, Saeed; Mohler, James L.; Attwood, Kristopher; Liu, Qian; Gutierrez, Camille; Toussi, Mehdi Seilanian

doi:10.1097/as9.0000000000000292

Cited by 7 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most often, motion data from surgical videos or kinematic data from robotic systems or sensors were collected from simulators rather than during actual surgical procedures. The most common simulators used were robotic box models ( n = 27, 54%) [ 13 , 14 , 16 – 20 , 22 , 23 , 25 , 29 , 32 , 37 , 43 , 45 – 48 , 50 , 53 – 56 , 58 – 61 ]. Laparoscopic simulators were the second most common setting for data collection ( n = 15, 30%) [ 12 , 21 , 24 , 26 , 27 , 30 , 31 , 35 , 36 , 40 , 42 , 49 , 51 , 54 , 57 ].…”

Section: Resultsmentioning

confidence: 99%

“…Four different types of input data were used throughout the 50 studies: video data ( n = 25, 50%) [ 12 , 13 , 15 , 21 , 24 , 25 , 27 , 28 , 31 – 34 , 36 , 38 , 39 , 41 – 45 , 50 – 52 , 55 , 58 ], kinematic data ( n = 22, 44%) [ 14 , 16 – 20 , 22 , 23 , 29 , 35 , 37 , 40 , 46 – 49 , 54 , 56 , 57 , 59 – 61 ], eye tracking data ( n = 2) [ 36 , 53 ], and functional near-infrared spectroscopy (fNIRS) data ( n = 2) [ 26 , 30 ]. Video recordings either from laparoscopic/robotic cameras or external cameras are used in 25 studies (50%).…”

Section: Resultsmentioning

confidence: 99%

“…The most common performance metric reported in the studies included in this systematic review is accuracy ( n = 35, 70%) [ 12 , 14 , 15 , 18 – 20 , 22 – 26 , 29 – 33 , 35 – 37 , 39 , 41 , 43 , 46 – 51 , 53 – 55 , 57 – 60 ]. Accuracies of the best performing models range between 0.58 and 1.…”

Section: Resultsmentioning

confidence: 99%

“…Accuracies of the best performing models range between 0.58 and 1. Other performance metrics reported include F1-score ( n = 9) [ 25 , 29 , 32 , 40 , 50 , 51 , 53 , 54 , 60 ], recall also known as sensitivity ( n = 11, 22%) [ 12 , 18 , 25 , 26 , 32 , 35 , 50 , 51 , 53 , 54 , 60 ], specificity ( n = 4) [ 14 , 26 , 44 , 45 ], and AUC-ROC ( n = 4) [ 14 , 29 , 44 , 45 ]. Six studies [ 16 , 21 , 27 , 28 , 42 , 56 ] did not report a performance metric at all.…”

Section: Resultsmentioning

confidence: 99%

See 3 more Smart Citations

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

et al. 2023

View full text Add to dashboard Cite

Background Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery. Methods A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of the included studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. Results In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB. Conclusion AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Preprocessing involved applying a moving average filter with a window size of 3 points to reduce noise, and a velocity-threshold identification fixation filter with a threshold of 30° per second to identify fixation and saccadic points. Twelve eye gaze features were extracted, including pupil diameter, entropy, fixation time points, saccade time points, gaze direction change, and pupil trajectory length for both eyes [ 17 , 18 ].…”

Section: Methodsmentioning

confidence: 99%

Performance and learning rate prediction models development in FLS and RAS surgical tasks using electroencephalogram and eye gaze data and machine learning

Shafiei,

Shadpour,

Intes

et al. 2023

Surg Endosc

Self Cite

View full text Add to dashboard Cite

Objective This study explored the use of electroencephalogram (EEG) and eye gaze features, experience-related features, and machine learning to evaluate performance and learning rates in fundamentals of laparoscopic surgery (FLS) and robotic-assisted surgery (RAS). Methods EEG and eye-tracking data were collected from 25 participants performing three FLS and 22 participants performing two RAS tasks. Generalized linear mixed models, using L1-penalized estimation, were developed to objectify performance evaluation using EEG and eye gaze features, and linear models were developed to objectify learning rate evaluation using these features and performance scores at the first attempt. Experience metrics were added to evaluate their role in learning robotic surgery. The differences in performance across experience levels were tested using analysis of variance. Results EEG and eye gaze features and experience-related features were important for evaluating performance in FLS and RAS tasks with reasonable results. Residents outperformed faculty in FLS peg transfer (p value = 0.04), while faculty and residents both excelled over pre-medical students in the FLS pattern cut (p value = 0.01 and p value < 0.001, respectively). Fellows outperformed pre-medical students in FLS suturing (p value = 0.01). In RAS tasks, both faculty and fellows surpassed pre-medical students (p values for the RAS pattern cut were 0.001 for faculty and 0.003 for fellows, while for RAS tissue dissection, the p value was less than 0.001 for both groups), with residents also showing superior skills in tissue dissection (p value = 0.03). Conclusion Findings could be used to develop training interventions for improving surgical skills and have implications for understanding motor learning and designing interventions to enhance learning outcomes. Graphical abstract

show abstract

Classification of subtask types and skill levels in robot-assisted surgery using EEG, eye-tracking, and machine learning

Shafiei,

Shadpour,

Mohler

et al. 2024

Surg Endosc

View full text Add to dashboard Cite

Background Objective and standardized evaluation of surgical skills in robot-assisted surgery (RAS) holds critical importance for both surgical education and patient safety. This study introduces machine learning (ML) techniques using features derived from electroencephalogram (EEG) and eye-tracking data to identify surgical subtasks and classify skill levels. Method The efficacy of this approach was assessed using a comprehensive dataset encompassing nine distinct classes, each representing a unique combination of three surgical subtasks executed by surgeons while performing operations on pigs. Four ML models, logistic regression, random forest, gradient boosting, and extreme gradient boosting (XGB) were used for multi-class classification. To develop the models, 20% of data samples were randomly allocated to a test set, with the remaining 80% used for training and validation. Hyperparameters were optimized through grid search, using fivefold stratified cross-validation repeated five times. Model reliability was ensured by performing train-test split over 30 iterations, with average measurements reported. Results The findings revealed that the proposed approach outperformed existing methods for classifying RAS subtasks and skills; the XGB and random forest models yielded high accuracy rates (88.49% and 88.56%, respectively) that were not significantly different (two-sample t-test; P-value = 0.9). Conclusion These results underscore the potential of ML models to augment the objectivity and precision of RAS subtask and skill evaluation. Future research should consider exploring ways to optimize these models, particularly focusing on the classes identified as challenging in this study. Ultimately, this study marks a significant step towards a more refined, objective, and standardized approach to RAS training and competency assessment.

show abstract

Developing Surgical Skill Level Classification Model Using Visual Metrics and a Gradient Boosting Algorithm

Cited by 7 publications

References 39 publications

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

Performance and learning rate prediction models development in FLS and RAS surgical tasks using electroencephalogram and eye gaze data and machine learning

Classification of subtask types and skill levels in robot-assisted surgery using EEG, eye-tracking, and machine learning

Contact Info

Product

Resources

About