Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

Abstract

The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance-such as scenarios, raters, and items-and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

Evaluating validity evidence for 2 instruments developed to assess students' surgical skills in a simulated environment

“…67 Studies in veterinary and nursing education have reported successfully using fewer than 20 persons in conjunction with larger number of conditions per facet. 52,68 The small sample size may have contributed to the observed low variance in student scores as assessed by the checklist. If the surgical skills examination on a model, scored using the checklist, is to be used as a high-stakes assessment, particularly at other institutions, further validity evidence and additional reliability data should be gathered to maintain a solid validity argument for its use.…”

“…While specific guidelines on minimal sample size for generalizability studies have not been established, a minimum of 20 persons for a 1 facet design has been suggested 67 . Studies in veterinary and nursing education have reported successfully using fewer than 20 persons in conjunction with larger number of conditions per facet 52,68 . The small sample size may have contributed to the observed low variance in student scores as assessed by the checklist.…”

Consistency of examiner judgements in medical education: a case study of competency-based assessment

“…G Theory simultaneously facilitates the identification of multiple sources of variance and quantification of each individual variance in a measurement (Bloch & Norman, 2012;Briesch, Swaminathan, Welsh, & Chafouleas, 2014). Based on a variance component analysis (Brennan, 2010), it differentiates the sources of variance and estimates the magnitude of each variance (O'Brien, Thompson, & Hagler, 2017).…”

“…biases and poorly-constructed marking criteria (Prion et al, 2016). The major limitation of using CTT in reliability analyses is that it focuses on only a single source of error variance at any one time (Bloch & Norman, 2012), as it does not differentiate sources of error variance (O'Brien et al, 2017).…”

“…One way of quantifying different sources of variance is to estimate the variance components of each facet (e.g., students, examiners, and stations), that is, the amount of variance associated with each facet or interaction between facets (Shavelson & Webb, 1991). Based on a variance component analysis (Brennan, 2010), G Theory differentiates the sources of variance and estimates the magnitude of each source of variance (O'Brien et al, 2017). It explores the similarity of the examiners' scores to any other raw scores that the examiners might give under identical circumstances and develops a reliability analysis, as well as monitoring the overall quality of an OSCE (Iramaneerat, Yudkowsky, Myford, & Downing, 2008).…”