Introduction of Patient Video Clips into Computer-Based Testing: Effects on Item Statistics and Reliability Estimates

Abstract

The inclusion of patient video clips in computer-based testing is feasible from technical, practical, and psychometric perspectives. Further study is needed to gather validity evidence for this novel question format.

Towards a conceptual framework demonstrating the effectiveness of audiovisual patient descriptions (patient video cases): a review of the current literature

“…The absence of a gold standard measure to assess training interventions may have led researchers to be opportunistic in their use of outcome measures. In this review seven studies gave no justification for the outcome measure used [13,15,25,26,29,30]. In addition comments by the authors themselves on limitations to the outcome measures were absent in five of the studies [13,26,30,31].…”

“…In this work both learner knowledge and learner satisfaction were assessed by different measures (a video test, a written test and a course evaluation). Three other studies [14,25,31] had more than one outcome measure, although these were all subtle variations on a theme such as scores in different types of clinical examination in the same test.…”

Performance and views of examiners in the Applied Knowledge Test for the nMRCGP licensing examination

“…Further evaluation could include views of candidates, such as those with disabilities, 21 and the use of different formats, for example video clips, 22 case simulations, 23 computer-based adaptive testing (in which test content changes depending on the candidate's responses), 24 and other stimulus formats. These might include video, audio, and multimedia clips that could increase validity of the assessment and open up the potential to test in new areas appropriate to the scope of computer-based testing.…”

Objective Structured Video Examinations (OSVEs) for Geriatrics Education

“…In particular, the recurring costs of paying standardized patients and the administrative costs of managing objective structured clinical examinations (OSCEs) typically prohibit the use of this assessment approach in residency education. However, with the emergence of low-cost digital video the opportunity to use video triggers in lieu of live standardized patients provides an alternative to the traditional OSCE approach for assessing learner performance (Lieberman, Frye, Litwins, Rasmusson, & Boulet, 2003;Hulsman, Mollema, Hoos, Haes, & Donnison-Speijer, 2004;Vlantis, Lee, & Hasselt, 2004).…”

“…A cost-effective simulation variant is to require the resident to assess the presence or absence of key history, physical exam and/ or other diagnostic, therapeutic and/or long-term management variables (Lieberman et al, 2003) in response to a video trigger depicting the clinical experience. Compared with the traditional, observation-based assessment of the learner in an actual clinical setting, the benefit of video-based simulation is that the faculty member can standardize the assessment by controlling all the major sources of variance inherent in the clinical setting that contribute to measurement error (e.g., complexity of the patient's problem, the patient's level of competence, time available, different or untrained faculty observers) except the resident's performance.…”