Subject bias and the retrospective pretest in retrospect

Abstract

(Pieter Koele, Sponsor) Studies employing designs including a thentest were reviewed to examine to what extent they demonstrated a response shift, while controlling for subject bias by including a placebo control condition. A computer search was executed to detect articles that included the Howard, Ralph, et al. (1979) article in their reference list. Of 24 studies, 16 did not warrant any statement about the occurrence of a response-shift bias, relative to a subject bias, because they lacked a placebo control condition. Support for the subject bias hypothesis, relative to the response-shift hypothesis, was found in one of eight studies that did include a placebo control condition. With respect to these results, three supplementary considerations were taken into account. It was concluded that subject bias has not proved to be an unqualified alternative explanation of the response-shift interpretation in the reported studies.The research design most frequently employed to evaluate training effectiveness is the two-group pretestposttest design (Campbell & Stanley, 1966, Design 4) . Cronbach and Furby (1970)noted that for pretest and posttest scores to be comparable, a common metric must exist for the two sets of scores. In using self-report instruments, Howard, Ralph, et al. (1979)found that a response shift-a change in internal standard from pretest to posttest-renders pretest and posttest scores incompatible. To obtain selfreport measures of pretest and posttest levels of functioning with respect to the same internal standard, Howard, Ralph, et al. (1979) administered a retrospective pretest-a thentest-at the time of posttesting. Subjects were asked to report how they then perceived themselves to have been prior to the intervention. A significant mean difference between pre-and thenscores in the experimental condition would indicate a response shift . Since a response shift is dependent on the treatment, it should occur in the experimental and not in the control condition. If a response shift is demonstrated, a valid estimate of a treatment effect is given by the mean difference between post-and thenscores (Howard, Ralph, et al. , 1979). However, designs including a thentest may be more susceptible to subject bias resulting from subjects' perception of the demand characteristics of the experimental situation (Orne, 1969) than Design 4 of Campbell and Stanley (1966). Since the thentest is administered at the same time as the posttest, subjects may, in an attempt to indicate favorable results, adjust their thenscores in a downward direction. Consequently, a mean pre/then difference score may not reflect a shift in internal standard, but may represent subject bias instead.

Controlling response shift bias: the use of the retrospective pre‐test design in the evaluation of a master's programme

“…The theory of response shift would state that this conceptual shift occurred due to exposure to the educational programme during which students became aware of their ability and were able to accurately reconceptualise where they were at the beginning of the programme following completion of the programme. The argument underlying the use of a retrospective pretest is that that scores obtained from posttest minus thentest are more likely to accurately reflect a positive intervention effect than scores obtained from the traditional pretest-posttest method (Howard 1980, Sprangers 1988, 1989a, 1989b, Sprangers and Hoogstraten 1987, 1988a, 1988b1989, 1991.…”

“…To control for response shift bias it has been suggested that the retrospective pretest method (other terms used in the literature include the then-post design, thentest, or the post-then-pre design) be used in self-report measures of change (Howard et al 1979, Howard 1980, Bray et al 1984, Sprangers and Hoogstraten 1987, 1988a, 1988b, 1989, 1991, Sprangers 1988, 1989a, 1989b, Goedhart and Hoogstraten 1992, Umble et al 2000, Rohs 2002. The retrospective pretest method differs from the traditional pretest-posttest design in that both posttest and pretest perceptions of respondents are collected at the same time.…”

“…Basically the design asks the respondent to recall a point in the past and compare it to where they are now. The collection of thentest and posttest ratings at the same time leads to the reduction of response-bias due to the fact that the respondent is making the ratings at time one (thentest) and time two (posttest) from the same perspective (Howard 1980, Sprangers 1988, 1989a, 1989b. The theoretical assumption underlying the retrospective pretest method is that by asking the respondent to rate where there are now in terms of ability in relation to the construct under investigation and where they were prior to the educational intervention, they will be using the same internal frame of reference or metric to rate the construct of interest.…”

Innovations in Measurement: Visual Analog Scales and Retrospective Pretest Self-Report Designs

“…In a similar vein, the conventional pretest-posttest self-report measurement design is by far one of the most popular methods to collect and analyze change data (Sprangers, 1989). After an individual participated in a treatment or experience, it is important to estimate the "true change" which occurred in the individual (Cronbach & Furby, 1970).…”

Evaluating learning in library training workshops: using the retrospective pretest design

“…Researchers have generally concluded that differences between retrospective pretests and conventional posttests are adequate indicators of change in behaviours [7][8][9][10][11][12] The purpose of this study was to assess the effectiveness of an evaluation instrument using the retrospective pretest to measure changes in participants' behaviour after library training.…”