Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment

Siau, Keith; Dunckley, Paul; Valori, Roland; Feeney, Mark; Hawkes, Neil; Anderson, John T.; Beales, Ian; Wells, C; Thomas-Gibson, Siwan; Johnson, Gavin

doi:10.1055/a-0576-6667

Cited by 43 publications

(60 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It follows the standard DOPS format of deconstructing a procedure into constituent steps (items), enveloped within broader groupings (domains). The gastroscopy DOPS was modified in July 2016 to include change to the 4-point scoring scale from a performance-based to a supervision-based (entrustment) scale [8], refinement of items and descriptors (following further working group review), and a generic domain for assessing endoscopic non-technical skills (ENTS). The updated gastroscopy DOPS contains 34 items, split into six domains, with each item accompanied by a detailed descriptor (Supplementary Fig.…”

mentioning

confidence: 99%

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

et al. 2019

Self Cite

View full text Add to dashboard Cite

Background Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. Methods This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. Results In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing 'pre-procedure', 'technical', and 'post-procedure non-technical' skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of 'pre-procedure' (100-124 procedures), 'technical' (150-174 procedures), 'post-procedure non-technical' skills (200-224 procedures), and global competency (225-249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. Conclusion This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.

show abstract

mentioning

confidence: 99%

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…One area is the development of an assessment tool for NTS specific to the endoscopic setting. Although some existing tools incorporate NTS-related items, 41,42 there is no tool with strong validity evidence that is targeted solely towards endoscopic NTS. 43 Furthermore, a qualitative exploration of endoscopists', endoscopy nurses', and patients' perceptions of NTS would be valuable to inform future interventions.…”

Section: Discussionmentioning

confidence: 99%

Non‐technical skills curriculum incorporating simulation‐based training improves performance in colonoscopy among novice endoscopists: Randomized controlled trial

Walsh

Scaffidi

Khan

et al. 2020

Digestive Endoscopy

View full text Add to dashboard Cite

Background and Aims Non‐technical skills (NTS), involving cognitive, social and interpersonal skills that complement technical skills, are important for the completion of safe and efficient procedures. We investigated the impact of a simulation‐based curriculum with dedicated NTS training on novice endoscopists’ performance of clinical colonoscopies. Methods A single‐blinded randomized controlled trial was conducted at a single center. Novice endoscopists were randomized to a control curriculum or a NTS curriculum. The control curriculum involved a didactic session, virtual reality (VR) simulator colonoscopy training, and integrated scenario practice using a VR simulator, a standardized patient, and endoscopy nurse. Feedback and training were provided by experienced endoscopists. The NTS curriculum group received similar training that included a small‐group session on NTS, feedback targeting NTS, and access to a self‐reflective NTS checklist. The primary outcome was performance during two clinical colonoscopies, assessed using the Joint Advisory Group Direct Observation of Procedural Skills (JAG DOPS) tool. Results Thirty‐nine participants completed the study. The NTS group (n = 21) had superior clinical performance during their first (P < 0.001) and second clinical colonoscopies (P < .0.001), compared to the control group (n = 18). The NTS group performed significantly better on the VR simulator (P < 0.05) and in the integrated scenario (P < 0.05). Conclusion Our findings demonstrate that dedicated NTS training led to improved performance of clinical colonoscopies among novices.

show abstract

“…6 In general, current evidence showed that workplace based assessment is not yet sufficient in terms of reliability but highly valid. [12][13][14][15][16] DOPS is not widely used in undergraduate dental education in Pakistan, but in light of findings of current study DOPs may enhance learning capacity of procedural skills in orthodontics at undergraduate level. However there are certain disadvantages of DOPS, such as, financial cost, time cost, and training of trainers and trainees and feasibility issues.…”

Section: Discussionmentioning

confidence: 99%

The Effect of Applying DOPS on BDS Students’ Orthodontic Skills: A Quasi-Experimental Study

Azeem¹,

Iqbal²,

Haq³

2019

J Pak Dent Assoc

View full text Add to dashboard Cite

OBJECTIVE: To compare the effectiveness of two teaching methods i.e. Directly Observed Procedural Skills (DOPS) and traditional teaching of procedural skills in Orthodontics to the BDS dental students. METHODOLOGY: Present study was conducted on 30 BDS final year students of Dental Section-Faisalabad Medical University. Dental students were divided into two groups i.e. 15 students in the DOPS group and 15 students in the traditional teaching group. The Multiple choice questions (MCQs) tests were taken before and after the procedural skills teaching from both the groups. The cumulative scores were calculated before and after the teaching of procedural skills in both the groups. Student paired t test was used for comparison of data. RESULTS: Results showed that for all the taught orthodontic procedural skills, post-test scores were found to be significantly higher in DOPS group in comparison with the post-test scores in traditional teaching group. (p >0.05) CONCLUSION: The teaching of orthodontic procedural skills to BDS students is more effective using DOPS than the traditional teaching of procedural skills.

show abstract

Changes in scoring of Direct Observation of Procedural Skills (DOPS) forms and the impact on competence assessment

Abstract: Endoscopy assessors applied a greater range of scores using the new DOPS scale based on degree of supervision in two cohorts of trainees matched for experience. Our study provides construct validity evidence in support of the new scale format.

Cited by 43 publications

References 10 publications

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

Non‐technical skills curriculum incorporating simulation‐based training improves performance in colonoscopy among novice endoscopists: Randomized controlled trial

The Effect of Applying DOPS on BDS Students’ Orthodontic Skills: A Quasi-Experimental Study

Contact Info

Product

Resources

About