ERCP assessment tool: evidence of validity and competency development during training

Abstract

Background The endoscopic retrograde cholangiopancreatography (ERCP) direct observation of procedural skills (DOPS) is a 27-item competency assessment tool that was developed to support UK ERCP training. We evaluated validity of ERCP DOPS and competency development during training. Methods This prospective study analyzed ERCP DOPS performed in the UK between July 2016 and October 2018. Reliability was measured using Cronbach’s alpha, and DOPS scores were benchmarked using the contrasting groups method. The percentage of competent scores was averaged for each item, domain, and overall rating, and stratified by lifetime procedure count to evaluate learning curves. Multivariable analyses were performed to identify predictors of DOPS competence. Results 818 DOPS (109 trainees, 80 UK centers) were analyzed. Overall Cronbach’s alpha was 0.961. Attaining competency in 87 % of assessed DOPS items provided the optimal competency benchmark. This was achieved in the domain sequence of: pre-procedure, post-procedure management, endoscopic non-technical skills, cannulation & imaging, and execution of selected therapy, and across all items after 200 – 249 procedures (89 %). After 300 procedures, the benchmark was reached for selective cannulation (89 %), but not for stenting (plastic 73 %; metal 70 %), sphincterotomy (80 %), and sphincteroplasty (56 %). On multivariable analysis, lifetime procedure count (P = 0.002), easier case difficulty (P < 0.001), trainee grade (P = 0.03), and higher lifetime DOPS count (P = 0.01) were predictors of DOPS competence. Conclusion This study provides novel validity, reliability, and learning curve data for ERCP DOPS. Trainees should have a minimum of 300 hands-on ERCP procedures before undertaking summative assessment for independent practice.

Impact of COVID-19 on endoscopy trainees: an international survey

“…32,33 Second, determination of competence should rely less on attaining minimum numbers and more on the use of objective and validated methods of competency assessment. This is best achieved through the use of objective performance tools with strong validity evidence, such as the Assessment of Competency in Endoscopy, 34 Direct Observation of Procedural Skills, [35][36][37] and the Gastrointestinal Endoscopy Competency Assessment Tool, 38 which can allow trainers to target feedback provision in a formative manner and to benchmark global competence for summative signoff. 35 Despite these measures, it may be necessary for some trainees to extend their endoscopy training.…”

Curriculum for ERCP and endoscopic ultrasound training in Europe: European Society of Gastrointestinal Endoscopy (ESGE) Position Statement

“…A systematic review in 2015 included nine studies overall but, in the five looking at selective cannulation, the range of procedure volumes required was 79-300 [43]. In 2019 Siau et al reported from the UK on the outcomes of formative ERCP assessment from the nationwide electronic training portfolio and demonstrated that the competency benchmark for selective cannulation of 89 % was only achieved after 300 procedures [28]. Also in 2019, Wani et al reported on the learning curves of 62 advanced endoscopy trainees and concluded that the average trainee required 250 cases to achieve core skills in ERCP, and 305 cases for the more complex Grade 2 cases [44].…”

Response

“…2 Cognitive skills (eg, pathology recognition, intra-/postprocedural management) are integral to competent endoscopy and can take longer to acquire than technical skills. 4,5 COVID-19 has catalyzed the uptake of previously underused cognitive-based digital resources (Figs. 1 and 2), which are well placed to address cognitive skills.…”