Objective Assessment of Manual Skills and Proficiency in Performing Epidural Anesthesia—Video-Assisted Validation

Abstract

Background and Objectives: Demand is growing for objective assessment of manual skills and competencies of invasive procedures. The aim of this study was to validate an objective tool for assessing residents' skill in performing epidural anesthesia by use of a global assessment scale and a 3-scale, 27-stage checklist. We wish to demonstrate that this tool can differentiate operators with different levels of training.Methods: Second-year anesthesia residents were recruited. Their previous experience was assessed by questionnaire. They were repeatedly videotaped performing epidural anesthesia over a 6-month period. Videotaping was done in a blinded manner that masked the identity and level of training of the residents. Three blinded, independent examiners evaluated each session by use of a specifically devised assessment tool that consisted of a global rating scale and a 3-scale, 27-stage checklist to judge the skill level and grade the videotaped sessions.Results: Twenty-one sessions by 6 residents were videotaped over 6 months. Interrater reliability for the different checklist and global-rating form items shows moderate to high degree of agreement for most stages. Total scores demonstrate almost perfect agreement (/ICC Ϯ SE ϭ 0.90 Ϯ 0.03 and 0.83 Ϯ 0.13, respectively; P Ͻ .0001) between examiners. To test whether higher total scores are associated with greater experience, a series of repeated-measures ANCOVAs were performed. In both the global-rating form and the checklist, a significant relation between total scores and epidurals done was found to exist (checklist: P Ͻ .0001; global rating: P Ͻ .0001). Conclusions:The results of our study show that scores on a system that consists of a global-rating form and a task-specific checklist had a significant relation to the number of epidural insertions performed (i.e., experience). The interrater reliability of these assessment tools was very strong. Evaluation of technical skills by an objective tool under direct observation, as opposed to laboratory setting, may create a more reliable standard of assessment. Furthermore, residency programs could use these evaluations to identify deficiencies in teaching programs and trainees who require extra instruction. Reg Anesth Pain Med 2006;31:304-310.

Validation of the Imperial College Surgical Assessment Device (ICSAD) for labour epidural placement

“…The examiners evaluated each performance using a task-specific checklist and a GRS, both of which had been validated in previous studies. 2,6 Both examiners were trained in the use of the checklist and GRS for the grading of epidural anesthesia procedures. One of the examiners had previous experience grading performances using the checklist and GRS for a prior study.…”

“…We hypothesized that the ICSAD would be able to discriminate between operators of different levels of experience in performing a labour epidural in all three ICSAD dexterity scores: path length, number of movements, and time, thus establishing construct validity for assessment of this skill. Furthermore, we postulated that there would be a correlation between the three ICSAD dexterity scores and a previously validated task-specific checklist 2 and GRS, 1,2 therefore establishing concurrent validity for the ICSAD measurements. 6 The epidural for labour analgesia was examined because of its importance in the field of anesthesiology and its particular complexity as a technical skill for anesthesiologists.…”

“…1 There has been a recent trend towards establishing more objective measures of technical skill in medicine. 2 Direct observation with criteria, such as checklists and global rating scales, may improve the assessment of technical skill by reducing observer bias. 3 Within anesthesiology, construct validity has been established for a variety of task-specific checklists and global rating scales.…”

“…[1][2][3] Unlike checklists, global rating scales (GRS) allow for consideration of aspects of manual dexterity, such as respect for tissue, flow of procedure, movement efficiency, and instrument handling. The GRS was recently demonstrated within anesthesia as being capable of distinguishing between operators with various levels of experience in performing labour epidurals and interscalene nerve blocks.…”

“…The GRS was recently demonstrated within anesthesia as being capable of distinguishing between operators with various levels of experience in performing labour epidurals and interscalene nerve blocks. 1,2 Despite the addition of objective criteria for evaluation, both the task-specific checklist and the global rating scale retain a degree of subjectivity. Although checklists and GRSs are widely used for technical skill assessment in various specialties, they remain in their infancy for use in technical skill assessment and have not been incorporated into formative and summative evaluations.…”