Generalizability Theory: An Introduction With Application to Simulation Evaluation

Prion, Susan; Gilbert, Gregory E.; Adamson, Katie Anne

doi:10.1016/j.ecns.2016.08.006

Cited by 10 publications

(18 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As an illustration, medical education is one setting where G theory has been widely used (e.g., Prion et al, 2016). For example, when nursing students are examined on clinical skills, they might encounter ten structured clinical problems (or stations) that they are required to respond to in the presence of an examiner who rates their performance on a 1-10 scale for each problem.…”

Section: Generalizability Theorymentioning

confidence: 99%

Distinguishing transient versus stable aspects of depression in New Zealand Pacific Island children using Generalizability Theory

Paterson

Medvedev

Sumich

et al. 2018

Journal of Affective Disorders

View full text Add to dashboard Cite

show abstract

Section: Generalizability Theorymentioning

confidence: 99%

Distinguishing transient versus stable aspects of depression in New Zealand Pacific Island children using Generalizability Theory

Paterson

Medvedev

Sumich

et al. 2018

Journal of Affective Disorders

View full text Add to dashboard Cite

show abstract

“…G theory provides researchers with more information to better inform decisions regarding assessments. Two very approachable introductions to G theory are Prion et al and Bloch and Norman [22, 23].…”

Section: Methodsmentioning

confidence: 99%

Psychometric testing of a checklist for procedural training of peripheral intravenous insertion

Buckley

Gilbert²,

Bauman³

2019

Adv Simul

Self Cite

View full text Add to dashboard Cite

Background Nurses, medical technologists, nuclear medicine technologists, pre-hospital providers, and medical students are a few groups of healthcare learners asked to learn intravenous (IV) cannulation in their training (J Surg Educ. 69:536–43, 2012). Despite the fact that IV cannulation has been taught to several health professions, it is difficult to find a psychometrically validated checklist to guide teaching this skill in the simulated procedural training (Pediatrics 124: 610-9, 2009, J Assoc Vasc Access 21: 196-204, 2016). In the absence of a pragmatic, valid checklist for the initial teaching of peripheral IV skills in the simulation procedural skills lab, this investigation sought to describe the process and create a psychometrically valid checklist. Methods Expert raters used Lawshe’s method for identifying valid items from the universe of items for IV insertion. Gwet’s AC 2 and generalizability (G) theory was used assess inter-rater reliability. Results The literature and in-house IV checklists were examined for steps to inserting a peripheral IV, and the steps were compiled into a survey and sent to experts who rated each item. Of the 37 potential steps, 16 steps were identified as being psychometrically valid. The checklist content validity index was .82. Inter-rater reliability was .94 (95% CI .91–.98). Good inter-rater reliability was confirmed using generalizability theory. Conclusions This study created and provided evidence of content validity and reliability for this checklist using Lawshe’s methodology. As such, this method of evaluating a checklist for validity and reliability evidence can be followed for other healthcare checklists. This checklist can be used for teaching IV placement in healthcare students in the simulation procedural training lab. Electronic supplementary material The online version of this article (10.1186/s41077-019-0092-y) contains supplementary material, which is available to authorized users.

show abstract

“…Although attention to reliability and validity is increasingly being reported in the literature, often only coefficient a or interrater reliability statistics are provided to satisfy reliability testing, and usually only vague references are made to experts ensuring content validity. To our knowledge, only one article has described the use of G theory to identify a minimum number of scenarios or minimum number of raters to achieve high reliability in observation-based assessment in nursing (Prion, Gilbert, & Haerling, 2016). On the other hand, studies conducted in medical education using standardized patients and HPSs have successfully utilized G theory to determine the number of scenarios and number of raters needed for reasonable reliability estimates (Boulet & Murray, 2010;Boulet et al, 2003).…”

Section: Generalizability Theorymentioning

confidence: 99%

Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

2017

View full text Add to dashboard Cite

The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance-such as scenarios, raters, and items-and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

show abstract

Generalizability Theory: An Introduction With Application to Simulation Evaluation

Cited by 10 publications

References 15 publications

Distinguishing transient versus stable aspects of depression in New Zealand Pacific Island children using Generalizability Theory

Distinguishing transient versus stable aspects of depression in New Zealand Pacific Island children using Generalizability Theory

Psychometric testing of a checklist for procedural training of peripheral intravenous insertion

Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

Contact Info

Product

Resources

About