Preparing and Presenting Validation Studies

Abstract

Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

Mixed methods instrument validation: Evaluation procedures for practitioners developed from the validation of the Swiss Instrument for Evaluating Interprofessional Collaboration

“…As a complement to cognitive interviewing, our procedures have the benefit of detecting issues with question design that might have been missed in the smaller sample cognitive interviews. The procedures we developed and cognitive interviewing can both be situated within the Messick validity framework [27,48], which applies generally to instrument validation and is independent of any MM validation frameworks. Viewed from the perspective of Messick's framework, our procedures should aim to ascertain whether the questions were understood as intended [49] in order to minimise unwanted variability and provide response process evidence [48].…”

“…The procedures we developed and cognitive interviewing can both be situated within the Messick validity framework [27,48], which applies generally to instrument validation and is independent of any MM validation frameworks. Viewed from the perspective of Messick's framework, our procedures should aim to ascertain whether the questions were understood as intended [49] in order to minimise unwanted variability and provide response process evidence [48]. We elaborate on the elements we take from the literature review in the methods section "Validation Criteria and Procedures.…”

“…Adaptations were focused on making questions clearer by adding definitions [56] and underlining keywords to emphasise key aspects [61,62]. These changes are unlikely to fundamentally change the instrument's psychometric properties, but will rather help to reduce Grand-Guillaume-Perrenoud et al BMC Health Services Research (2023) 23:83 unwanted variability [48]. This has provided a refined instrument which can be retested for further psychometric evaluation.…”

Development of peer assessment rubrics in simulation-based learning for advanced cardiac life support skills among medical students

“…In validating the rubric, the study employed Messick's validity framework to guide and ensure the validity of this research [21][22][23]. The framework outlines a systematic method to obtain construct validity evidence, with Messick highlighting five essential aspects: (1) content, ensuring alignment of test items with the intended construct; (2) response process, prioritizing data integrity and clear instructions; (3) internal structure, examining the exam's psychometric properties; (4) relations with other variables, analyzing theoretical correlations; and (5) consequences, determining effects on learners, instructors, and the system [21].…”