Statistical testing in treatment by replication designs: Three options reconsidered

M > 1

“…(p. 575) That the standard choice should be to treat sampled materials, such as replications, as random seems clear (see, e.g., Clark, 1973;Coleman, 1964Coleman, , 1979Cornfield & Tukey, 1956;Fontanelle, Phillips, & Lane, 1985;Forster & Dickinson, 1976;Jackson, Brashers, & Massey, 1992;Jackson & Jacobs, 1983;Keppel, 1991;Richter & Seay, 1987;Santa, Miller, & Shaw, 1979;Wickens & Keppel, 1983). Instead of the nested factor B/A (words within word type), we would have factor B (blocks of words) which crosses with factor A.…”

“…Evidence that effects of interest may be sensitive to the particular message samples used in experiments is reviewed in detail by Jackson, OKeefe, Jacobs, and Brashers (1989) and by Jackson, Brashers, and Massey (1992). The replaceability of replications is evident from the fad that different studies of the same phenomenon are not generally restricted to some particular persuasive message and by the fact that generalization is almost never restricted to the particular materials used in a study.…”

Variability of persuasive message effects

“…There has been substantial discussion of the choice between fixed-effects and random-effects (or 'mixed-model') analyses of replicated designs in primary research on com munication effects (e.g., Burgoon, Hall, &Pfau, 1991;Hunter, Hamilton, & Allen, 1989;Jackson, 1992;Jackson, Brashers, & Massey, 1992;Jackson, O'Keefe, Jacobs, & Brashers, 1989). These alternative statistical analyses differ in how they handle the presence of message replications in an experimental design.…”

Changing Conceptions of "Message Effects" A 24-Year Overview

“…Jackson and Jacobs (1983) made three recommendations for change in research practices connected to a distinction between the treatment of interest and the concrete message materials used to embody it: first, use of multiple messages in experiments using concrete messages to represent abstract message categories or to carry abstract message treatments; second, treatment of individual messages as a source of uncontrolled variation in estimates of the treatment effect; and third, attention to the quality of the individual messages as a sample from the category of interest. These themes have been elaborated and refined through vigorous debate in HCR (Jackson, O'Keefe, & Jacobs, 1988;Morley, 1988) and other journals of the field (M. Burgoon, Hall, & Pfau, 1991;Hunter, Hamilton, & Allen, 1989;Jackson, Brashers, &Massey, 1992;Jackson, OKeefe, &Brashers, 1994;Jackson, OKeefe, Jacobs, & Brashers, 1989;Slater, 1991), spawning additional reflections on design and analysis options (Jackson, 1991(Jackson, , 1992(Jackson, ,1993Jackson & Brashers, l993,1994a, 1994b. The subsequent writings have made clear that the arguments made in 1983 apply equally to other experimental components (such as the use of confederates).…”

“…When replications have been included as a strategy for permitting generalization, one way of thinking about what it means to generalize is to assume that what is wanted is a conclusion about an abstract message quality whose possible implementations are merely sampled by the specific stimuli included in the experiment. This line of thinking has suggested to many that replications should be treated not as fixed levels of an explanatory factor but as levels of a random factor that contributes sample-related variance to the size of the effects of theoretical interest (see Bonge, Schuldt, & Harper, 1992;Clark, 1973;Coleman, 1964Coleman, , 1979Crits-Christoph & Mintz, 1991;Fontenelle, Phillips, & Lane, 1985;Jackson, 1992;Jackson & Brashers, 1994b;Jackson et al, 1992;Maxwell & Bray, 1986;Richter & Seay, 1987;Santa, Miller, & Shaw, 1979;Wickens & Keppel, 1983;Zucker, 1990). Jackson and Brashers (1994a) noted that respondents are treated as random because they are a source of unpredictable variation in an experiment's outcome and argued that "analogous reasoning applied to replications would suggest that replications be treated as random if they are recognized as a source of unsystematic variation in experimental outcomes and if they are replaceable in principle with other replications of like kind" (p. 359).…”