A comparison study of pre/post-test and retrospective pre-test for measuring faculty attitude change

Abstract

We report on our investigation of a retrospective pre-test to measure faculty attitude change towards the use of active learning after the Physics and Astronomy New Faculty Workshop (NFW). The purpose of the study is to explore alternative methods of evaluating the effectiveness of educational interventions aimed at attitude change. In the current study, we focus on faculty attitudes that would support change in teaching practice. Using traditional pre/post surveys, we find that only knowledge of and skill using active learning are substantively increased by the workshop. We administered a retrospective pre-test, where participants retrospectively rate their pre-workshop attitudes on the post-workshop survey. The rationale for this approach is that participants do not start with a common understanding of what "active learning" entails, and the workshop provides a normalizing experience so participants shift their understanding of active learning (termed response shift bias) as well as potentially generating gains in positive attitudes towards active learning. Using the retrospective pre-test, we see attitudinal gains for most items, but pre-test and retrospective pre-test results are poorly and inconsistently correlated. Preliminary interviews are suggestive of response shift bias, but only for some items. We can conclude that the validity of pre-workshop attitude ratings is questionable, but because of a conflation of response shift bias with other reporting biases (such as social desirability) and respondent characteristics, further research is needed to indicate whether retrospective pre-testing is an improved approach.

Investigating the impact of an online hydration intervention in care homes using the COM-B model: a mixed methods study

“…Retrospective pre assessment of self-perceived knowledge also has the potential to decrease internal validity due to the impact of the implicit theory of change, in which participants believe pre-test scores should be lower than post-test [31]. Response bias can also occur where social desirability results in the participant feeling compelled to give the response they believe is expected [32]. These potential biases make a direct comparison for the outcomes of both studies difficult, however both face to face and online training appear to be effective in increasing hydration care self-perceived knowledge of care home staff.…”

Investigating the impact of an online hydration intervention in care homes using the COM-B model: a mixed methods study

“…Retrospective pre assessment of knowledge also has the potential to decrease internal validity due to the impact of the implicit theory of change, in which participants believe pre-test scores should be lower than post-test [29]. Response bias can also occur where social desirability results in the participant feeling compelled to give the response they believe is expected [30]. These potential biases make a direct comparison for the outcomes of both studies di cult, however both face to face and online training appear to be effective in increasing hydration care knowledge of care home staff.…”