Validation Evidence using Generalizability Theory for an Objective Structured Clinical Examination

Abstract

Objectives: Performance-based assessments, including objective structured clinical examinations (OSCEs), are essential learning assessments within pharmacy education. Because important educational decisions can follow from performance-based assessment results, pharmacy colleges/schools should demonstrate acceptable rigor in validation of their learning assessments. Though G-Theory has rarely been reported in pharmacy education, it would behoove pharmacy educators to, using G-Theory, produce evidence demonstrating reliability as a part of their OSCE validation process. This investigation demonstrates the use of G-Theory to describes reliability for an OSCE, as well as to show methods for enhancement of the OSCE’s reliability. Innovation: To evaluate practice-readiness in the semester before final-year rotations, third-year PharmD students took an OSCE. This OSCE included 14 stations over three weeks. Each week had four or five stations; one or two stations were scored by faculty-raters while three stations required students’ written responses. All stations were scored 1-4. For G-Theory analyses, we used G_Strings and then mGENOVA. Critical Analysis: Ninety-seven students completed the OSCE; stations were scored independently. First, univariate G-Theory design of students crossed with stations nested in weeks (p x s:w) was used. The total-score g-coefficient (reliability) for this OSCE was 0.72. Variance components for test parameters were identified. Of note, students accounted for only some OSCE score variation. Second, a multivariate G-Theory design of students crossed with stations (p· x s°) was used. This further analysis revealed which week(s) were weakest for the reliability of test-scores from this learning assessment. Moreover, decision-studies showed how reliability could change depending on the number of stations each week. For a g-coefficient >0.80, seven stations per week were needed. Additionally, targets for improvements were identified. Implications: In test validation, evidence of reliability is vital for the inference of generalization; G-Theory provided this for our OSCE. Results indicated that the reliability of scores was mediocre and could be improved with more stations. Revision of problematic stations could help reliability as well. Within this need for more stations, one practical insight was to administer those stations over multiple weeks/occasions (instead of all stations in one occasion).

Development of peer assessment rubrics in simulation-based learning for advanced cardiac life support skills among medical students

“…Previous research has used the generalizability theory to determine the variance in divergent instructional and learning models, identifying potential error sources in measurement conditions across the universe of observations [ 41 ]. Also, similar studies focusing on performance-based assessments with scoring rubrics frequently reported significant unexplained residual variance [ 36 , 41 , 46 ]. To the best of our knowledge, this study was the first to use generalizability theory analysis to assess ACLS simulation-based scenario scores among preclinical students with student raters.…”

Association between Clinical Case Validation and Students’ Success Rate in Virtual Pharmacy Simulation

“…34,35 Factor analysis has been advocated as a standard tool to determine the Objective Structured Clinical Examination (OSCE) in a medical school. [36][37][38] CFA testing revealed that only three cases (1, 4, and 10) were fit for evaluation. This is mainly based on CFA but is not a good fit for authentication analysis using RMSEA, CFI, TLI, and SRMR.…”

Comprehensive assessment of reliability and validity for the clinical cases in simulated community pharmacy

“…Nevertheless, case scenario 2 is failed to succeed in both the test for exact fit (Dagnall et al, 2018) and various fit measures (Alavi et al, 2020). Factor analysis is already a known standard tool to determine the OSCE in a medical school (Chesser et al, 2004;Paul et al, 2017;Peeters, 2021). In the present study, the measurement of both verbal and non-verbal communication skills has been ruled out due to the difficulty in measuring it across the various stations, and this decision is consistent with the recent research (Piumatti et al, 2021).…”

“…Planning the sample size is a challenging criterion in validating clinical cases for OSCE due to the difference of opinion in sample size determination by the previous researchers (Abdi et al, 2019;Majumder et al, 2019;Lim et al, 2020;Almaghaslah et al, 2021;Al-Haqan et al, 2021;Peeters et al, 2021). In this regard, the study once again emphasizes that the sample size should be more than 15 for each item to ensure the fitness of the clinical cases for the OSCE simulation.…”

“…In vision 2030, Saudi Arabia has a clear plan to implement GPP in all undergraduate pharmacy curriculum (Almaghaslah et al, 2021). Objective structured clinical examinations (OSCE) were established already in various pharmacy schools across the world (Peeters et al, 2021) for assessing the professional competence of clinical pharmacists in a structured manner (Al-Haqan et al, 2021) and for undergraduate pharmacy students (Abdi et al, 2019;Lim et al, 2020).…”