Developing an objective assessment of surgical performance from operating room video and surgical imagery

Abstract

An unbiased, repeatable process for assessing operating room performance is an important step toward quantifying the relationship between surgical training and performance. Hip fracture surgeries offer a promising first target in orthopedic trauma because they are common and they offer quantitative performance metrics that can be assessed from video recordings and intraoperative fluoroscopic images. Hip fracture repair surgeries were recorded using a head-mounted point-of-view camera. Intraoperative fluoroscopic images were also saved. The following performance metrics were analyzed: duration of wire navigation, number of fluoroscopic images collected, degree of intervention by the surgeon's supervisor, and the tip-apex distance (TAD). Two orthopedic traumatologists graded surgical performance in each video independently using an Objective Structured Assessment of Technical Skill (OSATS). Wire navigation duration correlated with weeks into residency and prior cases logged. TAD correlated with cases logged. There was no significant correlation between the OSATS total score and experience metrics. Total OSATS score correlated with duration and number of fluoroscopic images. Our results indicate that two metrics of hip fracture wire navigation performance, duration and TAD, significantly differentiate surgical experience. The methods presented have the potential to provide truly objective assessment of resident technical performance in the OR.

Surgical Skill Can be Objectively Measured From Fluoroscopic Images Using a Novel Image-based Decision Error Analysis (IDEA) Score

“…It is likely that a supervising surgeon, often a more-senior resident, would offer helpful tips and direction to the residents to help them accurately place the guide wire and ensure that the patient has a satisfactory result. The degree to which a supervising surgeon intervenes was not quantified in this study, but when it was previously quantified, it was not found to correlate with surgeon case log or tip-apex distance [15]. A third limitation may be in how some of the errors are measured.…”

“…Given that the number of decision errors and the mean angle of the wire movement errors were independent of the tip-apex distance, we chose to combine these metrics into a composite score to better assess surgical skill. Taylor et al [15] implemented a similar metric when looking at hip wire navigation performance in the operating room and found that combining the tip-apex distance, number of fluoroscopic images, overall time, and the amount of intervention needed from a supervising surgeon had a moderate correlation (R 2 = 0.43; p = 0.01) when related to the number of hip wire navigation cases logged by a resident. When combining the tip-apex distance, number of decision errors, and mean angle of decision errors into a composite metric, we observed a stronger correlation (Fig.…”

CORR Synthesis: How Have Film Review and Motion Analysis Been Used to Enhance Orthopaedic Surgical Performance?

“…After full-text review, we included 27 studies investigating the use of film review in orthopaedics including five randomized controlled trials [8, 9, 21, 36, 47] and 22 observational studies [3, 11, 12, 15, 20, 23, 24, 30, 41, 44, 50-52, 56, 61, 64, 65, 69, 77, 78, 81, 82]. We included 28 studies investigating the use of motion analysis in orthopaedics, including four randomized controlled trials [40, 58, 62, 70] and 24 observational studies [2, 7, 13, 14, 17, 18, 32, 38, 39, 43, 49, 57, 63, 67, 68, 71-73, 75, 76, 83, 84, 89, 90].…”

“…Blinded grading of surgical footage is overwhelmingly supported in previous research for its ability to differentiate among skill levels, with only one study [64] reporting difficulty differentiating above a certain skill threshold (Table 1). Surgical footage grading has been successful for basic surgical tasks such as suturing [12,77] and more advanced surgical procedures [11,20,24,41,50,51,52,65,69,78,81,82]. Surgical footage grading is also useful for tracking improvements in skill over time [3,61], and was found to be an effective method of grading arthroscopic Bankart repair skill with high inter-rater reliability [30].…”

Competency Versus Proficiency in Pediatric Orthopaedic Training

“…22 Video can be used in the operating room to improve reliability of proficiency assessments. 23 Such approaches can be successfully applied to non-operative treatments, such as Pavlik harness application. 24 Taken together, this supports the notion that teaching and measuring proficiency can be accomplished with structured education and assessment programs in the clinical environment.…”