Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards

Melchiors, Jacob; Petersen, Kristian Kjær; Todsen, Tobias; Bohr, Anne; Konge, Lars; Buchwald, Christian von

doi:10.1007/s00405-018-4971-y

Cited by 5 publications

(6 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For 39 of these studies, sufficient details on the variance components were reported or obtained from the authors for the quantitative synthesis (meta-analysis). [15][16][17][18][19][20][21][22][23][24][25][26][27][29][30][31][32][33][34][35][36][37][38][39][40][41][42][43]45,[47][48][49][50][51][52]54,[56][57][58]…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

et al. 2021

Self Cite

View full text Add to dashboard Cite

Purpose Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance. Method In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability. Results Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%–51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59–.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment. Conclusions G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills.

show abstract

Section: Resultsmentioning

confidence: 99%

“…15–58 For 39 of these studies, sufficient details on the variance components were reported or obtained from the authors for the quantitative synthesis (meta-analysis). 15–27,29–43,45,47–52,54,56–58…”

Section: Resultsmentioning

confidence: 99%

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Delphi method is well established to help experts achieve a consensus [24]. Several assessment tools for various procedures have been developed with this technique, and validity evidence has been established [25][26][27][34][35][36]. The electronic Delphi survey has some major advantages; panel anonymity ensures that participants can provide honest opinions without feeling pressured or vulnerable, and the electronic interface facilitates data collection and analysis.…”

Section: Discussionmentioning

confidence: 99%

Assessing competence in Chest Tube Insertion with the ACTION-tool: A Delphi study

Mol

Desender²,

Herzeele³

et al. 2022

International Journal of Surgery

Self Cite

View full text Add to dashboard Cite

“…27 It has successfully created other assessment tools found to be well supported by validity evidence. [28][29][30][31][32] The power of the methodology is the anonymized response process. All panelists' suggestions carry equal weight, and no individual(s) can dominate the decision process and bias results.…”

Section: Discussionmentioning

confidence: 99%

“…The Delphi method is a well-established consensus group strategy frequently used in medical education 27 . It has successfully created other assessment tools found to be well supported by validity evidence 28–32 . The power of the methodology is the anonymized response process.…”

Section: Discussionmentioning

confidence: 99%

Developing a tool to assess competence in resuscitative endovascular balloon occlusion of the aorta: An international Delphi consensus study

Engberg

Hörer

Rasmussen

et al. 2021

J Trauma Acute Care Surg

Self Cite

View full text Add to dashboard Cite

BACKGROUND:Resuscitative endovascular balloon occlusion of the aorta (REBOA) is an emergency procedure that is potentially lifesaving in major noncompressible torso hemorrhage. It may also improve outcome in nontraumatic cardiac arrest. However, the procedure can be technically challenging and requires the immediate presence of a qualified operator. Thus, evidence-based training and assessment of operator skills are essential for successful implementation and patient safety. A prerequisite for this is a valid and reliable assessment tool specific for the procedure. The aim of this study was to develop a tool for assessing procedural competence in REBOA based on best-available knowledge from international experts in the field. METHODS:We invited international REBOA experts from multiple specialties to participate in an anonymous three-round iterative Delphi study to reach consensus on the design and content of an assessment tool. In round 1, participants suggested items to be included.In rounds 2 and 3, the relevance of each suggested item was evaluated by all participants to reach consensus. Interround data processing was done systematically by a steering group. RESULTS:Forty panelists representing both clinical and educational expertise in REBOA from 16 countries (in Europe, Asia, and North and South America) and seven different specialties participated in the study. After 3 Delphi rounds and 532 initial item suggestions, the panelists reached consensus on a 10-item assessment tool with behaviorally anchored rating scales. It includes assessment of teamwork, procedure time, selection and preparation of equipment, puncture technique, guidewire handling, sheath handling, placement of REBOA catheter, occlusion, and evaluation. CONCLUSION:We present the REBOA-RATE assessment tool developed systematically by international experts in the field to optimize content validity. Following further studies of its validity and reliability, this tool represents an important next step in evidence-based training programs in REBOA, for example, using mastery learning.

show abstract

Procedure-specific assessment tool for flexible pharyngo-laryngoscopy: gathering validity evidence and setting pass–fail standards

Abstract: Based on our work, we recommend the use of the FLEXPAT in clinical assessment of FPL and the application of a pass-fail level of 72% for proficiency.

Cited by 5 publications

References 23 publications

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

Use of Generalizability Theory for Exploring Reliability of and Sources of Variance in Assessment of Technical Skills: A Systematic Review and Meta-Analysis

Assessing competence in Chest Tube Insertion with the ACTION-tool: A Delphi study

Developing a tool to assess competence in resuscitative endovascular balloon occlusion of the aorta: An international Delphi consensus study

Contact Info

Product

Resources

About