On Orthopedic Surgical Skill Prediction—The Limited Value of Traditional Testing

Putnam, Matthew D.; Kinnucan, Elspeth; Adams, Julie E.; Heest, Ann E. Van; Nuckley, David J.; Shanedling, Janet

doi:10.1016/j.jsurg.2014.11.001

Cited by 19 publications

(11 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Putnam et al [13] previously reported on the lack of correlation between traditional assessments of orthopaedic resident knowledge (Orthopaedics In-Training Exam [OITE] overall score, OITE trauma score, and an Objective Structured Assessment of Technical Skill [OSATS] score) and the structural integrity of fixation in the extraarticular fracture model that was used in the current study. That prior study tested only a limited number of residents (n = 15), and it included only PGY-3 (n = 8) and PGY-4 (n = 7) residents.…”

Section: Discussionmentioning

confidence: 76%

“…Sixth, we did not include inter-or intraobserver reliability testing of OSATS scoring as part of our experimental design. Prior work has established the general reliability of the OSATS approach to the rating of competencies [3,11,15], and our prior experience with the specific OSATS scoring schemes used [6,13,16,18] has shown consistency across the relatively few expert raters who have participated.…”

Section: Discussionmentioning

confidence: 99%

“…The second simulation involved fixing a simulated extraarticular distal radius fracture in a cadaveric specimen containing a hand, wrist, forearm, and elbow [13]. Identical extraarticular osteotomies of the distal radii were performed using a jig designed to remove 1 cm of metaphyseal bone immediately proximal to the proximal margin of the distal radioulnar joint (Fig.…”

Section: Study 2: Mechanical Construct Stability In An Extraarticularmentioning

confidence: 99%

See 2 more Smart Citations

Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

Anderson

Long

Thomas

et al. 2016

Clinical Orthopaedics &Amp; Related Research

Self Cite

View full text Add to dashboard Cite

Background Performance assessment in skills training is ideally based on objective, reliable, and clinically relevant indicators of success. The Objective Structured Assessment of Technical Skill (OSATS) is a reliable and valid tool that has been increasingly used in orthopaedic skills training. It uses a global rating approach to structure expert evaluation of technical skills with the experts working from a list of operative competencies that are each rated on a 5-point Likert scale anchored by behavioral descriptors. Given the observational nature of its scoring, the OSATS might not effectively assess the quality of surgical results. Questions/purposes (1) Does OSATS scoring in an intraarticular fracture reduction training exercise correlate with the quality of the reduction? (2) Does OSATS scoring in a cadaveric extraarticular fracture fixation exercise correlate with the mechanical integrity of the fixation?Methods Orthopaedic residents at the University of Iowa (six postgraduate year [PGY]-1s) and at the University of Minnesota (seven PGY-1s and eight PGY-2s) undertook a skills training exercise that involved reducing a simulated intraarticular fracture under fluoroscopic guidance. Iowa residents participated three times during 1 month, and Minnesota residents participated twice with 1 month between their two sessions. A fellowship-trained orthopaedic traumatologist rated each performance using a modified OSATS scoring scheme. The quality of the articular reduction obtained was then directly measured. Regression analysis was performed between OSATS scores and two metrics of articular reduction quality: articular surface deviation and estimated contact stress. Another skills training exercise involved fixing a simulated distal radius fracture in a cadaveric specimen. Thirty residents, distributed across four PGY classes (PGY-2 and PGY-3, Clinical Orthopaedics and Related Research ®A Publication of The Association of Bone and Joint Surgeons® n = 8 each; PGY-4 and PGY-5, n = 7 each), simultaneously completed the exercise at individual stations. One of three faculty hand surgeons independently scored each performance using a validated OSATS scoring system. The mechanical integrity of each fixation construct was then assessed in a materials testing machine. Regression analysis was performed between OSATS scores and two metrics of fixation integrity: stiffness and failure load. Results In the intraarticular fracture model, OSATS scores did not correlate with articular reduction quality (maximum surface deviations: R = 0.17, p = 0.25; maximum contact stress: R = 0.22, p = 0.13). Similarly in the cadaveric extraarticular fracture model, OSATS scores did not correlate with the integrity of the mechanical fixation (stiffness: R = 0.10, p = 0.60; failure load: R = 0.30, p = 0.10). Conclusions OSATS scoring methods do not effectively assess the quality of the surgical result. Efforts must be made to incorporate assessment metrics that reflect the quality of the surgical result. Clinical Relevance New objectiv...

show abstract

Section: Discussionmentioning

confidence: 76%

Section: Discussionmentioning

confidence: 99%

Section: Study 2: Mechanical Construct Stability In An Extraarticularmentioning

confidence: 99%

See 1 more Smart Citation

Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

Anderson

Long

Thomas

et al. 2016

Clinical Orthopaedics &Amp; Related Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…One other, more global approach to assessing surgical skills in a clinical setting is to systematically observe the objective elements of a surgeon's behavior. Currently, surgical task-specific checklists and OSATS are two commonly used techniques for measuring surgical skills; however, studies demonstrated that OSATS scores do not always correlate well with the outcomes [18, 19]. In a study of simulated distal radius fracture fixation, Putnam et al found that the participants' OSATS scores had no correlation with the actual mechanical strength of the fixation they performed [19].…”

Section: Overcoming Barriers To the Integration Of Simulation-basementioning

confidence: 99%

“…Currently, surgical task-specific checklists and OSATS are two commonly used techniques for measuring surgical skills; however, studies demonstrated that OSATS scores do not always correlate well with the outcomes [18, 19]. In a study of simulated distal radius fracture fixation, Putnam et al found that the participants' OSATS scores had no correlation with the actual mechanical strength of the fixation they performed [19]. Therefore, to be able to tie simulation-based surgical skills training to OR performance and surgical results, meticulously defined procedure-specific tasks should be assessed using standard measurement protocols that combine more than one objective technique, such as OSATS, motion analysis, and direct objective metrics, such as the accuracy of reduction, time to skill completion, and strength of a fracture fixation construct (Figure 1) [16].…”

Section: Overcoming Barriers To the Integration Of Simulation-basementioning

confidence: 99%

Advancing Simulation-Based Orthopaedic Surgical Skills Training: An Analysis of the Challenges to Implementation

Ateşok

Hurwitz

Anderson

et al. 2019

Advances in Orthopedics

View full text Add to dashboard Cite

Simulation-based surgical skills training is recognized as a valuable method to improve trainees’ performance and broadly perceived as essential for the establishment of a comprehensive curriculum in surgical education. However, there needs to be improvement in several areas for meaningful integration of simulation into surgical education. The purpose of this focused review is to summarize the obstacles to a comprehensive integration of simulation-based surgical skills training into surgical education and board certification and suggest potential solutions for those obstacles. First and foremost, validated simulators need to be rigorously assessed to ensure their feasibility and cost-effectiveness. All simulation-based courses should include clear objectives and outcome measures (with metrics) for the skills to be practiced by trainees. Furthermore, these courses should address a wide range of issues, including assessment of trainees’ problem-solving and decision-making abilities and remediation of poor performance. Finally, which simulation-based surgical skills courses will become a standard part of the curriculum across training programs and which will be of value in board certification should be precisely defined. Sufficient progress in these areas will prevent excessive development of training and assessment tools with duplicative effort and large variability in quality.

show abstract

Fluoroscopic image‐based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Mattioli,

Thomas,

Long

et al. 2023

Journal Orthopaedic Research

View full text Add to dashboard Cite

Psychomotor skill and decision‐making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image‐based behavior analysis of operating room (OR) performance can predict performer experience level (R2 = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2 = 0.59). However, it is unclear how objective image‐based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image‐based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired‐comparison survey of seven experts done based upon eight OR fluoroscopic wire navigation sequences shows that the experts’ preferences are best explained by objective metrics that reflect psychomotor and decision‐making behaviors which are counter‐productive to successful implant placement, like image count (R2 = 0.83) and behavior tally (R2 = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: the fluoroscopic image‐based analysis composite score (R2 = 0.40), expert consensus (R2 = 0.76), and performer experience (R2 = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. However, while expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2 = 0.11).This article is protected by copyright. All rights reserved.

show abstract

On Orthopedic Surgical Skill Prediction—The Limited Value of Traditional Testing

Cited by 19 publications

References 16 publications

Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

Advancing Simulation-Based Orthopaedic Surgical Skills Training: An Analysis of the Challenges to Implementation

Fluoroscopic image‐based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Contact Info

Product

Resources

About