Generalizing About Messages: Suggestions for Design and Analysis of Experiments

Abstract

Current research practices in communication create problems for both internal and external validity. One serious design flaw, which involves use of a single message to represent a category of messages, occurs in nearly all of the experimental research on communication effects. The problem with such a design is that an observed difference between categories may reflect only differences between individual, idiosyncratic cases. A related error, the “language‐as‐fixed‐effect fallacy,” involves use of several replications of each category, but analysis of the cases as fixed effects. The consequence is that findings cannot be generalized beyond the sample used. Future research should use multiple cases within each message category studied and treat cases as nested random effects. Since the cases cannot be true random samples from message populations, care must be taken to avoid known sources of bias or other barriers to generalization.

Ad framing effects for consumption products: An affect priming process

“…The final factor pertained to the featured attributes: comfort and style. Including the two outcome conditions and the two product attribute conditions was meant to increase generalizability of the findings and to reduce any idiosyncratic effects from featuring one particular type of psychological outcome or one particular product attribute (Jackson & Jacobs, 1983). Specific hypotheses were not proposed for these two factors because no effects were expected.…”

Messages as replications: Toward a message‐centered design strategy

“…Nor, for those cases requiring nested designs, do we dispute the need to treat messages as an explicit factor; on the contrary, we have consistently argued for treating messages as an explicit random factor, whether crossed with or nested within the levels of the message variable of interest. second bogus issue raised by HHA concerns the benefits to be gained from experimental manipulation of the message variables of interest. Jackson and Jacobs (1983) are misrepresented as favoring sampling of naturally-occurring messages over experimental manipulation of message features. Jackson and Jacobs did suggest collection of naturally-occurring messages as the basis for experimental manipulation, in order to respond to concerns about generalizability.…”

College Men’s Third-Person Perceptions about Idealized Body Image and Consequent Behavior

“…To increase the generalizability of the stimuli, we made sure that the ten pairs of advertisements covered a range of whole-body, half-body, and close-up shots of the models, and also included a significant range of brands and products (Chia 2007). The multiple pairs of models in the stimuli can also average out the confounding effects that might be produced by the idiosyncratic nature of a single pair of models (Jackson and Jacobs 1983).…”