Object detection to compute performance metrics for skill assessment in central venous catheterization

O'Driscoll, Olivia; Hisey, Rebecca; Camiré, Daenis; Erb, Jason; Howes, Daniel; Fichtinger, Gabor; Ungi, Tamás

doi:10.1117/12.2581889

Cited by 5 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This result means that our model can be used in a wider range of environments and facilities, where robotic surgery systems are not available for surgical trainees or faculty. Further, while some studies attempt to indirectly compute performance metrics for surgical skill [ 14 , 15 ], our model directly predicts performance on the GRS domains and provides the most pertinent assessment of surgical skill to trainees.…”

Section: Discussionmentioning

confidence: 99%

“…Instead of directly quantifying surgical performance, previous studies also focus on capturing proxies indicative of surgical performance, such as detecting surgical instruments [ 14 ], tracking instruments [ 15 ], or identifying events such as incisions [ 5 ]. Our work directly predicts the OSATS scores across five domains in a continuous regression framework.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

Kasa

Burns

Goldenberg

et al. 2022

Sensors

View full text Add to dashboard Cite

This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and time-series kinematic data. All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). This work is significant since it demonstrates that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. The study demonstrates an algorithm with state-of-the-art performance in surgical skill assessment. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

Kasa

Burns

Goldenberg

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…This result means that our model can be used in a wider range of environments and facilities, where robotic surgery systems are not available for surgical trainees or faculty. Further, while some studies attempt to indirectly compute performance metrics for surgical skill [9], [10], our model directly predicts performance on the GRS domains and provides the most pertinent assessment of surgical skill to trainees.…”

Section: Discussionmentioning

confidence: 99%

“…Although these advances in deep learning algorithms present the opportunity to automate some surgical technical skill assessments, previous research in this area has largely relied on classical machine learning algorithms leveraging engineered features in the data to classify performance, and thus far has largely been used to assess, global but not domain-specific, performance [4], [5], [6], [7], [8]. Deep learning has also been used successfully for object detection in skill assessment tasks [9], [10].…”

Section: Introductionmentioning

confidence: 99%

Multi-modal Deep Learning for Assessing Surgeon Technical Skill on a Surgical Knot-tying Task

Kasa¹,

Burns²,

Goldenberg³

et al. 2022

Preprint

View full text Add to dashboard Cite

<p>This paper introduces a new dataset of a surgical knot-tying task, and a multi-modal deep learning model that achieves comparable performance to expert human raters on this skill assessment task. Seventy-two surgical trainees and faculty were recruited for the knot-tying task, and were recorded using video, kinematic, and image data. Three expert human raters conducted the skills assessment using the Objective Structured Assessment of Technical Skill (OSATS) Global Rating Scale (GRS). We also designed and developed three deep learning models: a ResNet-based image model, a ResNet-LSTM kinematic model, and a multi-modal model leveraging the image and timeseries kinematic data. Results: All three models demonstrate performance comparable to the expert human raters on most GRS domains. The multi-modal model demonstrates the best overall performance, as measured using the mean squared error (MSE) and intraclass correlation coefficient (ICC). We found that multi-modal deep learning has the potential to replicate human raters on a challenging human-performed knot-tying task. As objective assessment of technical skill continues to be a growing, but resource-heavy, element of surgical education, this study is an important step towards automated surgical skill assessment, ultimately leading to reduced burden on training faculty and institutes.</p>

show abstract

“…One such computed-based assessment platform is Central Line Tutor developed by Hisey et al, which provides automatic skill assessment for central venous catheterization (CVC) [1]. Central Line Tutor relies on metrics such as path length and usage time to determine user skill [2]. Consistently low values of these metrics are associated with a higher skill level.…”

Section: Introductionmentioning

confidence: 99%

Using uncertainty quantification to improve reliability of video-based skill assessment metrics in central venous catheterization

Austin

Hisey

O'Driscoll

et al. 2023

Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling

View full text Add to dashboard Cite

Purpose: Computed-based skill assessment relies on accurate metrics to provide comprehensive feedback to trainees. Improving the accuracy of video-based metrics computed using object detection is generally done by improving the performance of the object detection network, however increasing its performance requires resources that cannot always be obtained. This study aims to improve the accuracy of metrics in central venous catheterization without requiring a high performing object detection network by removing false positive predictions identified using uncertainty quantification. Methods: The uncertainty for each bounding box was calculated using an entropy equation. The uncertainties were then compared to an uncertainty threshold computed using the optimal point of a Receiver Operating Characteristic curve. Predictions were removed if the uncertainty fell below the predefined threshold. 50 videos were recorded and annotated with ground truth bounding boxes. These bounding boxes were used to train an object detection network, which was used to produce predictive bounding boxes for the test set. This method was evaluated by computing metrics for the predictive bounding boxes with and without having false positives removed and comparing them to ground truth labels using a Pearson Correlation. Results: The Pearson Correlations for the baseline comparisons and the comparisons made using the results calculated using false positive removal were 0.922 and 0.816 for syringe path lengths, 0.753 and 0.510 for ultrasound path lengths, 0.831 and 0.489 for ultrasound usage times, and 0.857 and 0.805 for syringe usage times. Conclusions: This method consistently reduced inflated metrics, making it promising for improving metric accuracy.

show abstract

Object detection to compute performance metrics for skill assessment in central venous catheterization

Cited by 5 publications

References 8 publications

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

Multi-Modal Deep Learning for Assessing Surgeon Technical Skill

Multi-modal Deep Learning for Assessing Surgeon Technical Skill on a Surgical Knot-tying Task

Using uncertainty quantification to improve reliability of video-based skill assessment metrics in central venous catheterization

Contact Info

Product

Resources

About