Controlling response shift bias: the use of the retrospective pre‐test design in the evaluation of a master's programme

Drennan, Jonathan; Hyde, Abbey

doi:10.1080/02602930701773026

Cited by 147 publications

(120 citation statements)

References 28 publications

Supporting

Mentioning

109

Contrasting

Unclassified

Order By: Relevance

“…The tendency for the participant to redefine his/her frame of reference after training, called response-shift bias, has been found in numerous studies using pre and post assessments in a variety of training situations, and causes pretest-posttest comparisons to be inaccurate (Blome & Augistin, 2016;Davis, 2002;Drennan & Hyde, 2008;Howard, 1980;Metzoff, 1981b;Schwartz & Sprangers, 2010;Umble, Upshaw, Orton, & Matthews, 2000). Response-shift bias means that the participant has shifted his/her internal definition of the criteria by which to judge the appropriate rating of an item because the training has impacted his/her frame of reference (Blome & Augustin, 2016;Davis, 2002;Drennan & Hyde, 2008;Howard, 1980;Metzoff, 1981b;Schwartz & Sprangers, 2010;Umble et al, 2000).…”

Section: Methodsmentioning

confidence: 99%

“…Response-shift bias means that the participant has shifted his/her internal definition of the criteria by which to judge the appropriate rating of an item because the training has impacted his/her frame of reference (Blome & Augustin, 2016;Davis, 2002;Drennan & Hyde, 2008;Howard, 1980;Metzoff, 1981b;Schwartz & Sprangers, 2010;Umble et al, 2000).…”

Section: Methodsmentioning

confidence: 99%

“…Metzoff (1981a) identifies the two methods of measuring training effectiveness; the participant self-report questionnaire, and the behavioral observation measure. Using the traditional pre/post method for self-report assessments has long been considered the least effective and reliable method of measuring training outcomes because 1) on the pre-assessment participants may have limited knowledge of the construct being measured and therefore do not have the reference to adequately assess their prior knowledge, and 2) they tend to underestimate training benefits (Drennan & Hyde, 2008;Metzoff, 1981a;Rockwell & Kohn, 1989;Schwartz & Sprangers, 2010).…”

Section: Methodsmentioning

confidence: 99%

See 2 more Smart Citations

Co-Curricular Learning and Assessment in New Student Orientation at a Community College

Sandoval-Lucero¹,

Antony²,

Hepworth³

2017

View full text Add to dashboard Cite

With increased public pressure to impact student success, assess student learning and increase completion outcomes at community colleges, mandatory orientation can be an important first step on the path to success. Colleges also need to identify opportunities for assessing co-curricular learning in student affairs programs and functions. This study describes the process of implementing a mandatory new student orientation at a community college in Colorado. The program assessment included data collection at the individual student level to document student learning, and more broadly to demonstrate that the program contributed to increases in student retention outcomes and documented student learning from the beginning of the college transition process. The authors found statistically significant differences in student learning about key on boarding topics following orientation and the new student orientation also contributed to a seventeen percent increase in fall to spring retention of new students.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Co-Curricular Learning and Assessment in New Student Orientation at a Community College

Sandoval-Lucero¹,

Antony²,

Hepworth³

2017

View full text Add to dashboard Cite

show abstract

“…Yet, retrospective designs do have a few limitations that need to be mentioned. Drennan and Hyde (2008) noted how retrospective pretest self-report measures could suffer from social desirability and impression management. Demand characteristics and memory related problems have also been shown to influence the recall process in retrospective pretest selfreports (Pratt et al, 2000).…”

Section: Reasons To Use the Retrospective Pretest-posttest As Opposedmentioning

confidence: 99%

“…However, after participating in a program, participants have gathered more knowledge about the construct and are therefore more suited to give an accurate report. Yet, the response on the posttest will be answered from a different frame of reference compared against the pretest reference frame (Drennan & Hyde, 2008;Howard, Dailey, & Gulanick, 1979;Howard, 1980). The internal axes of reference for the pretest and the posttest differed within the individual when they completed the selfreport measures.…”

Section: Reasons To Use the Retrospective Pretest-posttest As Opposedmentioning

confidence: 99%

Innovations in Measurement: Visual Analog Scales and Retrospective Pretest Self-Report Designs

Gorrall

Curtis

Little

et al. 2016

View full text Add to dashboard Cite

Abstract. The authors discuss limitations of two popular measurement procedures, the Likert scale and conventional pretest-posttest self-report design. Both techniques have limits and yet are often combined, leading to restricted fidelity for measuring change. The authors go on to discuss two innovations in measurement that provide researchers with greater assessment fidelity: Visual analog scales and the retrospective pretest design. Moreover, when used in combination, these innovations measurement techniques provide dramatic increases in the power to detect and quantify change.Keywords. Attitudes, polytomous items, MRG, college students.Resumen. Se discuten las limitaciones de dos procedimientos de medición extremadamente populares; la escala Likert y los diseños pre-post tradicionales. Ambos métodos tienen limitaciones; estos métodos son comúnmente combinados obteniendo como resultado una fidelidad limitada para la medición del cambio. Se discute sobre dos innovaciones en medición que proveen al investigador una mayor fidelidad: escalas visuales análogas y el diseño pre-test retrospectivo. Además, cuando se usan de manera combinada, estas innovaciones en medición proveen notables incrementos en el poder de detección y la cuantificación de cambios.Palabras clave. Actitudes, items politómicos, MRG, estudiantes universitarios.

show abstract

Implementation of a North American pediatric emergency medicine simulation curriculum using the virtual resuscitation room

Leung

Foohey

Burns

et al. 2023

AEM Education and Training

View full text Add to dashboard Cite

Background Simulation provides consistent opportunities for residents to practice high‐stakes, low‐frequency events such as pediatric resuscitations. To increase standardization across North American residency programs, the Emergency Medicine Resident Simulation Curriculum for Pediatrics (EM ReSCu Peds) was developed. However, access to high‐quality simulation/pediatric expertise is not uniform. As the concurrent COVID‐19 pandemic necessitated new virtual simulation methods, we adapted the Virtual Resus Room (VRR) to teach EM ReSCu Peds. VRR is an award‐winning, low‐resource, open‐access distance telesimulation platform we hypothesize will be effective and scalable for teaching this curriculum. Methods EM residents completed six VRR EM ReSCu Peds simulation cases and received immediate facilitator‐led teledebriefing. Learners completed retrospective pre–post surveys after each case. Learners and facilitators completed end‐of‐day surveys. Primary outcomes were learning effectiveness measured by a composite of the Simulation Effectiveness in Teaching Modified (SET‐M) tool and self‐reported changes in learner comfort with case objectives. Secondary outcome was VRR scalability to teach EM ReSCu Peds using a composite outcome of net promoter scores (NPS), resource utilization, open‐text feedback, and technical issues. Results Learners reported significantly increased comfort with 95% (54/57) of EM ReSCu Peds–defined case objectives (91% cognitive, 9% psychomotor), with moderate (Cohen's d 0.71, 95% CI 0.67–0.76) overall effect size. SET‐M responses indicated simulation effectiveness, particularly with debriefing. Ninety EM residents from three North American residency programs were taught by 59 pediatric faculty from six programs over 4 days—more than possible if simulations were conducted in person. Learners (39) and faculty (68) NPS were above software industry benchmarks (13). Minor, quickly resolved, technical issues were reported by 18% and 29% of learners and facilitators, respectively. Conclusions Learners and facilitators report that the VRR is an effective and scalable platform to teach EM ReSCu Peds. This low‐cost, accessible distance simulation intervention could increase equitable, global access to high‐quality pediatric emergency education.

show abstract

Controlling response shift bias: the use of the retrospective pre‐test design in the evaluation of a master's programme

Cited by 147 publications

References 28 publications

Co-Curricular Learning and Assessment in New Student Orientation at a Community College

Co-Curricular Learning and Assessment in New Student Orientation at a Community College

Innovations in Measurement: Visual Analog Scales and Retrospective Pretest Self-Report Designs

Implementation of a North American pediatric emergency medicine simulation curriculum using the virtual resuscitation room

Contact Info

Product

Resources

About