A Comparison of Expert Ratings and Marker-Less Hand Tracking Along OSATS-Derived Motion Scales

Azari, David P.; Miller, Brady L.; Le, Brian; Greenberg, Jacob A.; Bruskewitz, Reginald C.; Long, Kristin L.; Chen, Guanhua; Radwin, Robert G.

doi:10.1109/thms.2020.3035763

Cited by 3 publications

(5 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neither computer-generated assessment was as accurate in the OR as they were on the benchtop simulations. As described in Azari, 1 both the motion economy assessment (slope = 0.65, intercept = 2.36, R 2 = 0.66) and fluidity of motion assessment (slope = 0.71, intercept = 1.98, R 2 = 0.77) exhibited reduced R 2 values when applied in the OR (0.49, 0.55, respectively), despite similar intercepts and slopes compared with the training data. The computer-generated assessment of fluidity of motion more closely mirrored the OR expert ratings than its motion economy counterpart.…”

Section: Resultsmentioning

confidence: 59%

“…When tested on the OR videos, the computer-generated assessments of motion economy and fluidity of motion derived from Azari 1 predicted the expert ratings moderately well, with mean squared errors of 1.74 and 1.58, respectively. For computer-generated assessment of motion economy (Fig.…”

Section: Resultsmentioning

confidence: 93%

“…The different methods in deriving a consensus score for OR videos (ie, consensus approach 2 ) and benchtop videos (ie, calibration, outlier removal 1 ) reduce consistency between the expert-generated and predicted ratings across the training and testing data. This challenge was mitigated by using the initial consensus ratings and video examples to calibrate all subsequent raters.…”

Section: Discussionmentioning

confidence: 99%

“…The features of hand motion (eg, speed, acceleration, etc) served as independent variables and were selected using common penalized regression shrinkage methods (eg, Ridge, LASSO, and Elastic Net) and balanced by examining the mean-square error and Akaike information criterion. Azari 1 further details the feature selection, refinement, and internal training of these models.…”

Section: Methodsmentioning

confidence: 99%

“…The purpose of this study is to test the accuracy of computer-generated assessments of surgical dexterity in the operating room (OR) during common suturing and tying tasks. The computer-generated assessments are previously trained linear and generalized additive models 1 that ingest a kinematic record of hand motion and produce measures along 2 previously tested 2 scales (Fig. 1) of motion economy (ie, efficiency, 0–10) and fluidity of motion (ie, smoothness, 0–10).…”

mentioning

confidence: 99%

See 4 more Smart Citations

Modeling Performance of Open Surgical Cases

Azari

Frasier²,

Miller³

et al. 2021

Sim Healthcare

Self Cite

View full text Add to dashboard Cite

Introduction: Previous efforts used digital video to develop computer-generated assessments of surgical hand motion economy and fluidity of motion. This study tests how well previously trained assessment models match expert ratings of suturing and tying video clips recorded in a new operating room (OR) setting Methods: Enabled through computer vision of the hands, this study tests the applicability of assessments born out of benchtop simulations to in vivo suturing and tying tasks recorded in the OR. Results: Compared with expert ratings, computer-generated assessments for fluidity of motion (slope = 0.83, intercept = 1.77, R 2 = 0.55) performed better than motion economy (slope = 0.73, intercept = 2.04, R 2 = 0.49), although 85% of ratings for both models were within ±2 of the expert response. Neither assessment performed as well in the OR as they did on the training data. Assessments were sensitive to changing hand postures, dropped ligatures, and poor tissue contact-features typically missing from training data. Computer-generated assessment of OR tasks was contingent on a clear, consistent view of both surgeon's hands. Conclusions: Computer-generated assessment may help provide formative feedback during deliberate practice, albeit with greater variability in the OR compared with benchtop simulations. Future work will benefit from expanded available bimanual video records.

show abstract

Section: Resultsmentioning

confidence: 59%

Section: Resultsmentioning

confidence: 93%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Modeling Performance of Open Surgical Cases

Azari

Frasier²,

Miller³

et al. 2021

Sim Healthcare

Self Cite

View full text Add to dashboard Cite

show abstract

An objective skill assessment framework for microsurgical anastomosis based on ALI scores

Gholami,

Manon,

Yao

et al. 2024

Acta Neurochir

View full text Add to dashboard Cite

Introduction The current assessment and standardization of microsurgical skills are subjective, posing challenges in reliable skill evaluation. We aim to address these limitations by developing a quantitative and objective framework for accurately assessing and enhancing microsurgical anastomosis skills among surgical trainees. We hypothesize that this framework can differentiate the proficiency levels of microsurgeons, aligning with subjective assessments based on the ALI score. Methods We select relevant performance metrics from the literature on laparoscopic skill assessment and human motor control studies, focusing on time, instrument kinematics, and tactile information. This information is measured and estimated by a set of sensors, including cameras, a motion capture system, and tactile sensors. The recorded data is analyzed offline using our proposed evaluation framework. Our study involves 12 participants of different ages ($$35.42\pm 9.78$$ 35.42 ± 9.78 years) and genders (nine males and three females), including six novice and six intermediate subjects, who perform surgical anastomosis procedures on a chicken leg model. Results We show that the proposed set of objective and quantitative metrics to assess skill proficiency aligns with subjective evaluations, particularly the ALI score method, and can effectively differentiate novices from more proficient microsurgeons. Furthermore, we find statistically significant disparities, where microsurgeons with intermediate level of skill proficiency surpassed novices in both task speed, reduced idle time, and smoother, briefer hand displacements. Conclusion The framework enables accurate skill assessment and provides objective feedback for improving microsurgical anastomosis skills among surgical trainees. By overcoming the subjectivity and limitations of current assessment methods, our approach contributes to the advancement of surgical education and the development of aspiring microsurgeons. Furthermore, our framework emerges to precisely distinguish and classify proficiency levels (novice and intermediate) exhibited by microsurgeons.

show abstract