The design and analysis of language experiments in communication

Statistical testing in treatment by replication designs: Three options reconsidered

“…As for the objections to the quasi F tests recommended by Clark, these tests do not figure among the choices confronted in treatment by replication designs, but if they did, it might be worth knowing that subsequent work has shown concerns about these tests to be grossly overblown (see, for example, Santa, Miller, & Shaw, 1979). Readers interested in keeping up-to-date on the language-as-fixed-effect fallacy, both pro and con, are advised to consult additional work by Coleman (1964Coleman ( ,1979, Fontanelle, Phillips, and Lane (1985), Forster and Dickinson (1976), Hunter, Hamilton, and Allen (1989), Jackson, O'Keefe, and Jacobs (1988), Jackson, O'Keefe, Jacobs, and Brashers (1989), Morley (1988), Richter and Seay (1987), Santa, Miller, and Shaw (1979), Slater (1991), Wickens and Keppel (1983), and Wike and Church (1976). 5 Subscripts denote effects as follows: "T" is treatments, "M" is messages, "T x M" is Treatment x Message interaction, "S/T" is subjects within treatments, and "S/TM" is subjects within Treatment x Message cells.…”

“…Apparently, Burgoon, Hall, and Pfau see replications as desirable, but not essential, to well-designed message effects research. As our purpose is to discuss what to do with replications once they have been included, we will not extend the argument for within-study replications, but refer interested readers to the directly-relevant colloquy between Hunter, Hamilton, and Allen (1989) and Jackson, O'Keefe, Jacobs, and Brashers (1989). 3 In an unreplicated treatment comparison, the question of how to treat the replications does not arise, as there is no replications factor.…”

“…An additional issue treated secondarily byBurgoon et al, and not at all in this essay, is the choice of what concrete message materials to use in manipulation of message variables; this issue has been discussed extensively elsewhere(Hunter, Hamilton, & Allen, 1989;Jackson, O'Keefe, & Jacobs, 1988;Jackson, O'Keefe, Jacobs, & Brashers, 1989;Morley, 1988).…”

Variability of persuasive message effects

“…There has been substantial discussion of the choice between fixed-effects and random-effects (or 'mixed-model') analyses of replicated designs in primary research on com munication effects (e.g., Burgoon, Hall, &Pfau, 1991;Hunter, Hamilton, & Allen, 1989;Jackson, 1992;Jackson, Brashers, & Massey, 1992;Jackson, O'Keefe, Jacobs, & Brashers, 1989). These alternative statistical analyses differ in how they handle the presence of message replications in an experimental design.…”

“…But questions have been raised about the adequacy of this research format for providing evidence supporting cross-message generalizations about message effects (see especially Jackson & Jacobs, 1983; for some subsequent discussion, see Hunter, Hamilton, & Allen, 1989;Jackson, 1992;Jackson, O'Keefe, & Brashers, 1994;Jackson, O'Keefe, & Jacobs, 1988;Jackson, O'Keefe, Jacobs, & Brashers, 1989;Morley, 1988;Slater, 1991). These concerns might be char acterized most broadly as connected with the possibility of variability in the effect of a given treatment across different messages.…”