Assessment of Competency-Based Behavioral Health Anticipatory Guidance Skills Among Pediatric Residents: the Role of Virtual Reality

Real, Francis J.; Whitehead, Monica; Rosen, Brittany L.; Meisman, Andrea; Crosby, Lori E.; Klein, Melissa; Herbst, Rachel Becker

doi:10.1007/s41347-021-00202-w

Cited by 6 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13 Our adapted instrument demonstrated high reliability when used by paediatric psychologists to assess residents' competencies during VR simulations. 9 Using this 'expert' instrument, a prior trial demonstrated enhanced BHAG and MI skills among residents that completed PRE-VENT training when compared to an untrained group. 12 Though the expert instrument provided extensive assessment data that was advantageous for research, its broad use by clinicians was limited due to its complexity.…”

Section: Approachmentioning

confidence: 99%

“…Residents' skills were assessed using an instrument adapted from the MI treatment integrity (MITI) instrument that has previously been used to facilitate learning and ensure fidelity in research studies 13 . Our adapted instrument demonstrated high reliability when used by paediatric psychologists to assess residents' competencies during VR simulations 9 . Using this ‘expert’ instrument, a prior trial demonstrated enhanced BHAG and MI skills among residents that completed PREVENT training when compared to an untrained group 12 .…”

Section: Approachmentioning

confidence: 99%

“…Virtual reality (VR) simulation allows learners to interact with graphical environments and characters (i.e., avatars) to deliberately practise skills in a safe learning environment. In addition to training, VR may serve as a platform on which to obtain validity evidence related to assessment tools 9 . Validity evidence refers to the extent to which interpretation of a learner's performance on a tool is justifiable to inform decisions 10 .…”

Section: Introductionmentioning

confidence: 99%

“…In addition to training, VR may serve as a platform on which to obtain validity evidence related to assessment tools. 9 Validity evidence refers to the extent to which interpretation of a learner's performance on a tool is justifiable to inform decisions. 10 Messick provides a framework for obtaining validity evidence that has been traditionally used in medical education.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Virtual reality informs clinical observation tool

Real

Ollberding

Sikora

et al. 2023

The Clinical Teacher

Self Cite

View full text Add to dashboard Cite

Background Assessing trainees' skills via workplace‐based assessments is challenging given the lack of psychometrically valid instruments. Our team previously developed an observation instrument to assess residents' competencies in behavioral health anticipatory guidance (BHAG) and motivational interviewing (MI) though its reliable usage required expert raters (e.g., paediatric psychologists). Our purpose was to establish validity evidence for a modified tool for clinicians. Approach This study utilised data from a prior educational trial of a virtual reality (VR)‐based behavioural health curriculum for paediatric residents. First, group interviews were conducted with clinicians to modify the expert instrument. Next, recorded VR simulations (n = 10) were scored to assess interrater reliability between clinicians. Finally, a physician used the tool to assess its ability to discriminate between residents' skill levels (n = 55). Evaluation Modifications during group interviews included a change from frequency counts for items related to MI to the binary outcome of present/absent and the addition of an entrustment item. On interrater‐reliability testing, the clinician tool demonstrated mostly substantial or near perfect agreement for items related to BHAG. MI items demonstrated a range of agreement. Using 55 recorded VR simulations, the clinician tool discriminated between trained versus untrained residents in BHAG skills (p = 0.002) and level of entrustment (p = 0.001). Differences between groups in MI adherence was not statistically significant (p = 0.095) as it was on the expert instrument. Implications VR demonstrated potential as a novel approach for obtaining validity evidence for tool development to support workplace‐based assessments. Further work assessing usage of the clinician tool in real‐world settings is warranted.

show abstract

Section: Approachmentioning

confidence: 99%