Use of a formal assessment instrument for evaluation of resident operative skills in pediatric neurosurgery

Abstract

<jats:sec> <jats:title>OBJECT</jats:title> <jats:p>Currently there is no standardized tool for assessment of neurosurgical resident performance in the operating room. In light of enhanced requirements issued by the Accreditation Council for Graduate Medical Education’s Milestone Project and the Matrix Curriculum Project from the Society of Neurological Surgeons, the implementation of such a tool seems essential for objective evaluation of resident competence. Beyond compliance with governing body guidelines, objective assessment tools may be useful to direct early intervention for trainees performing below the level of their peers so that they may be given more hands-on teaching, while strong residents can be encouraged by faculty members to progress to conducting operations more independently with passive supervision. The aims of this study were to implement a validated assessment tool for evaluation of operative skills in pediatric neurosurgery and determine its feasibility and reliability.</jats:p></jats:sec> <jats:sec> <jats:title>METHODS</jats:title> <jats:p>All neurosurgery residents completing their pediatric rotation over a 6-month period from January 1, 2014, to June 30, 2014, at the authors’ institution were enrolled in this study. For each procedure, residents were evaluated by means of a form, with one copy being completed by the resident and a separate copy being completed by the attending surgeon. The evaluation form was based on the validated Objective Structured Assessment of Technical Skills for Surgery (OSATS) and used a 5-point Likert-type scale with 7 categories: respect for tissue; time and motion; instrument handling; knowledge of instruments; flow of operation; use of assistants; and knowledge of specific procedure. Data were then stratified by faculty versus resident (self-) assessment; postgraduate year level; and difficulty of procedure. Descriptive statistics (means and SDs) were calculated, and the results were compared using the Wilcoxon signed-rank test and Student t-test. A p value &lt; 0.05 was considered statistically significant.</jats:p></jats:sec> <jats:sec> <jats:title>RESULTS</jats:title> <jats:p>Six faculty members, 1 fellow, and 8 residents completed evaluations for 299 procedures, including 32 ventriculoperitoneal (VP) shunt revisions, 23 VP shunt placements, 19 endoscopic third ventriculostomies, and 18 craniotomies for tumor resection. There was no significant difference between faculty and resident self-assessment scores overall or in any of the 7 domains scores for each of the involved residents. On self-assessment, senior residents scored themselves significantly higher (p &lt; 0.02) than junior residents overall and in all domains except for “time and motion.” Faculty members scored senior residents significantly higher than junior residents only for the “knowledge of instruments” domain (p = 0.05). When procedure difficulty was considered, senior residents’ scores from faculty members were significantly higher (p = 0.04) than the scores given to junior residents for expert procedures only. Senior residents’ self-evaluation scores were significantly higher than those of junior residents for both expert (p = 0.03) and novice (p = 0.006) procedures.</jats:p></jats:sec> <jats:sec> <jats:title>CONCLUSIONS</jats:title> <jats:p>OSATS is a feasible and reliable assessment tool for the comprehensive evaluation of neurosurgery resident performance in the operating room. The authors plan to use this tool to assess resident operative skill development and to improve direct resident feedback.</jats:p></jats:sec>

Neurosurgeon as educator: a review of principles of adult education and assessment applied to neurosurgery

“…Data were then stratified by faculty versus resident, postgraduate year level, and difficulty of procedure. 37 Although there were questions regarding the construct validity and reliability of this method, 48 the study did demonstrate the feasibility of using OSATS for neurosurgical resident evaluation.…”

“…Supervised, interpersonal mentorships developed through direct apprenticeship, coaching, and training in the operating room are a key component of neurosurgical training. 18,37,53 Unfortunately, opportunities to operationalize this type of teaching experience are increasingly difficult because of Accreditation Council for Graduate Medical Education (ACGME) resident duty-hour limitations, 41,44,58 financial pressures to be efficient in the operating room, 8,43,72 and the understandable emphasis on highquality care with low complication rates and short hospital stays. It is therefore imperative that surgical education also evolve to reflect an equal degree of efficiency and that effective methods for teaching skills outside the operative setting be employed.…”

“…77 These milestones are intended to be metrics by which residency programs can evaluate resident performance on a semiannual basis. 37 The milestones do not provide a standardized approach to evaluation of operative skills, nor is there any attempt to formulate how the learner acquires this knowledge base. 37,77 In response to these needs, the American Association of Neurological Surgeons (AANS) and the Congress of Neurological Surgeons (CNS) developed a Matrix Curriculum that defines the objective knowledge or technical skills for a given competency to determine the best teaching methods and assessment tools and sets a standard expectation for proficiency.…”

Endoscopic Management of Cavernous Carotid Surgical Complications: Evaluation of a Simulated Perfusion Model

“…1 Simulation training can increase proficiency and allow learners to reach milestones set forth by accrediting agencies. 2 Furthermore, simulation allows for appreciation of relevant neuropathology and anatomy that may be pertinent intraoperatively. 3 In particular, endoscopic skull base approaches, which have a steep learning curve, can be practiced in a controlled and replicable environment.…”

Learner-Driven Synthesis of Assessment Data: Engaging and Motivating Residents in Their Milestone-Based Assessments

“…Lower performing learners may not have insight into their low performance; conversely, higher performing learners may be more aware of their deficits than their strengths [11,13,14]. A few studies with very specific learner types and environments have shown good correlation between expert assessments and learner self-assessments, but these have been the exception rather than the rule [15][16][17][18][19]. Furthermore, the success reported in these environments may be due to a focus on self-monitoring rather than global self-assessment [20].…”