American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents’ Technical Skills

Shabahang, Mohsen; Adetunji, Adegoke A.; Daley, Brian; Mpinga, Ebondo; Sudan, Ranjan; Tillou, Areti; Blair, Patrice Gabler; Park, Yoon Soo; Lipsett, Pamela A.; Jarman, Benjamin T.; Sachdeva, Ajit K.

doi:10.1016/j.jsurg.2022.07.007

Cited by 2 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Statistical Considerationsmentioning

confidence: 99%

“…In designing an assessment tool, reporting the steps undertaken in the validation process is of great importance. Herein, we have described the process of validation and gathering validity evidence . Some of the statistical methods that can contribute to the validation process include the following:…”

Section: Statistical Considerationsmentioning

confidence: 99%

“…Herein, we have described the process of validation and gathering validity evidence. 6 Some of the statistical methods that can contribute to the validation process include the following: • Borderline regression or logistic regression (through receiver operating characteristic curves) is used to establish the cutoff scores. The borderline regression method is a linear regression model based on the examinee's scores that establishes a score above which the candidate is considered to be at an acceptable performance level.…”

Section: Statistical Considerationsmentioning

confidence: 99%

See 2 more Smart Citations

Practical Guide to Assessment Tool Development for Surgical Education Research

Shabahang,

Schwartz,

Feldman

2024

JAMA Surg

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Statistical Considerationsmentioning

confidence: 99%

Section: Statistical Considerationsmentioning

confidence: 99%

Section: Statistical Considerationsmentioning

confidence: 99%

See 1 more Smart Citation

Practical Guide to Assessment Tool Development for Surgical Education Research

Shabahang,

Schwartz,

Feldman

2024

JAMA Surg

Self Cite

View full text Add to dashboard Cite

show abstract

“…bjective assessment of technical skill in surgical training is an essential component of surgical education. Current assessment methods include expert observation, [1][2][3] objective skills assessment checklists, 4,5 and simulators. [6][7][8][9][10][11] Given the duty hour restrictions and increasing pressures for operating room efficiency, surgical trainees report increasingly insufficient assessment, feedback, and autonomy for independent practice in some fields.…”

mentioning

confidence: 99%

Pilot Analysis of Surgeon Instrument Utilization Signatures Based on Shannon Entropy and Deep Learning for Surgeon Performance Assessment in a Cadaveric Carotid Artery Injury Control Simulation

Balu,

Pangal,

Kugener

et al. 2023

Operative Neurosurgery

View full text Add to dashboard Cite

BACKGROUND AND OBJECTIVES: Assessment and feedback are critical to surgical education, but direct observational feedback by experts is rarely provided because of time constraints and is typically only qualitative. Automated, video-based, quantitative feedback on surgical performance could address this gap, improving surgical training. The authors aim to demonstrate the ability of Shannon entropy (ShEn), an information theory metric that quantifies series diversity, to predict surgical performance using instrument detections generated through deep learning. METHODS: Annotated images from a publicly available video data set of surgeons managing endoscopic endonasal carotid artery lacerations in a perfused cadaveric simulator were collected. A deep learning model was implemented to detect surgical instruments across video frames. ShEn score for the instrument sequence was calculated from each surgical trial. Logistic regression using ShEn was used to predict hemorrhage control success. RESULTS: ShEn scores and instrument usage patterns differed between successful and unsuccessful trials (ShEn: 0.452 vs 0.370, P < .001). Unsuccessful hemorrhage control trials displayed lower entropy and less varied instrument use patterns. By contrast, successful trials demonstrated higher entropy with more diverse instrument usage and consistent progression in instrument utilization. A logistic regression model using ShEn scores (78% accuracy and 97% average precision) was at least as accurate as surgeons' attending/resident status and years of experience for predicting trial success and had similar accuracy as expert human observers. CONCLUSION: ShEn score offers a summative signal about surgeon performance and predicted success at controlling carotid hemorrhage in a simulated cadaveric setting. Future efforts to generalize ShEn to additional surgical scenarios can further validate this metric.

show abstract

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents’ Technical Skills

Cited by 2 publications

References 17 publications

Practical Guide to Assessment Tool Development for Surgical Education Research

Practical Guide to Assessment Tool Development for Surgical Education Research

Pilot Analysis of Surgeon Instrument Utilization Signatures Based on Shannon Entropy and Deep Learning for Surgeon Performance Assessment in a Cadaveric Carotid Artery Injury Control Simulation

Contact Info

Product

Resources

About