A Power Struggle: Between- Vs. Within-Subjects Designs in Deductive Reasoning Research

Abstract

This experiment examined the relative merits of using within-and between-subjects designs to investigate deductive reasoning. Two issues were investigated: 1) the potential for expectancy and fatigue effects when using within-subjects designs, and 2) the relative power of within-vs between-subjects designs. Participants were presented with problems in a standard belief-bias paradigm in which the believability of putative conclusions varied orthogonally to their validity. The belief bias effect, as well as the effect of validity, and the interaction between beliefs and validity, were not affected by reasoners' expectations regarding the number of problems they had to solve. The effect of beliefs and the belief by validity interaction were only marginally affected by the number of problems solved, despite adequate power to observe an effect. Thus, neither expectancy nor fatigue appear to have affected performance, suggesting that there are few drawbacks to using a within-subjects design. In contrast, however, a power analysis clearly established the desirability of using within-relative to between-subjects designs. Withinsubjects designs require far fewer participants to detect effects of comparable size; this was especially true for higher-order (interaction) effects. Finally, we provide a power analysis of within-and between-subjects designs that should be of general utility to researchers planning studies using proportions as a dependent measure.Key words: belief bias, deductive reasoning, statistical power, expectancy effects, carryover effectsThe default method for investigating deductive reasoning is a within-subjects design in which participants are asked to solve a large number of problems in a single sitting

Effects of the Video-feedback intervention to promote positive parenting and sensitive discipline on mothers’ neural responses to child faces: A randomized controlled ERP study including pre- and post-intervention measures

“…In addition, the distribution of participants across the intervention and control group was skewed, with 44 participants in the control group and 22 in the intervention group. This limits statistical power, although not to an unacceptable level which is also due to the within-subject design (Thompson & Campbell, 2004;. Still, we recommend future studies to incorporate larger samples sizes existing of both mothers and fathers.…”

Developmental plasticity varied with sex and position in hatching hierarchy in nestlings of the asynchronous European roller, Coracias garrulus

“…In the present study, we used a between‐nest design to investigate variation in resource allocation within broods in relation to individual characteristics such as sex and hatching order, which is not easily achieved in a within‐nest design. Within‐subjects designs are more powerful to detect effects than between‐subjects designs (Thompson & Campbell, 2004). However, in the present study, a between‐subject design has the advantage of better allowing an identification of the effects of sex and position in the hatching hierarchy on nestling characteristics.…”

“…However, in the present study, a between‐subject design has the advantage of better allowing an identification of the effects of sex and position in the hatching hierarchy on nestling characteristics. Furthermore, by using this type of design, we avoided the carryover effects of within‐nest designs (Thompson & Campbell, 2004). These carryover effects comprise any effects that are transferred from one experimental condition to another and that might, for example, cause different behaviour in control and experimental siblings, thus creating a confounding extraneous variable that varies with the treatment.…”

Are Within-Subjects Designs "Transparent"?

“…Other concerns involve the possible presence of within-subjects context effects, such as practice, fatigue, expectancy, order and carry-over effects. Thompson and Campbell (2004), for instance, pointed out that most research on deductive reasoning is done within subjects and investigated whether between-subjects designs might be more appropriate due to the possibility of expectancy and fatigue effects in within-subjects studies.…”

“…This change, however, was very small and nonsignificant, thereby providing evidence that in research wherein participants solve eight reasoning problems back-toback (eight being the median number of problems used in research on deductive reasoning), fatigue effects will likely not jeopardize the interpretation of within-subjects designs. Thompson and Campbell (2004) concluded that within-subjects designs should be preferred due to the substantial increase in power they afford. Zimmerman (1997), for instance, has pointed out that even when a correlation is as small as .05, using a between-rather than within-subjects design significantly reduces power.…”