A cautionary note on the power of the test for the indirect effect in mediation analysis

Abstract

Recent simulation studies have pointed to the higher power of the test for the mediated effect vs. the test for the total effect, even in the presence of a direct effect. This has motivated applied researchers to investigate mediation in settings where there is no evidence of a total effect. In this paper we provide analytical insight into the circumstances under which higher power of the test for the mediated effect vs. the test for the total effect can be expected in the absence of a direct effect. We argue that the acclaimed power gain is somewhat deceptive and comes with a big price. On the basis of the results, we recommend that when the primary interest lies in mediation only, a significant test for the total effect should not be used as a prerequisite for the test for the indirect effect. However, because the test for the indirect effect is vulnerable to bias when common causes of mediator and outcome are not measured or not accounted for, it should be evaluated in a sensitivity analysis.

Helping motivation and well-being of chronic pain couples: a daily diary study

“…With regard to the ICP outcomes, the presence of the total effect (c) of partners' helping motivation on ICP outcomes was not a prerequisite for testing indirect effects. 29 Robustness of the mediated effects against unmeasured common causes (or confounders) of the mediator and outcome was assessed by means of sensitivity analyses. We found that for our mediated effects, relatively strong effects of such unmeasured time-varying common causes of M (ie, ICPs' need satisfaction/frustration) and Y (ie, different ICP outcomes) are needed to yield zero (or nonsignificant) mediated effects.…”

Mediating pathways and gender differences between shift work and subjective cognitive function

“…Although the indirect pathway between shift work and subjective cognitive function was statistically significant, the total effect was not, even though these estimates were both in the same direction. This may have resulted because the detection of significant indirect effects requires less study power and stronger assumptions about unmeasured confounding compared with total or direct effects 54. Finally, the cross-sectional study design limits our ability examine causal relationships.…”

Ironic effects of feedback on contingency of self-worth: Why self-reports of contingency are biased

“…First, although for both social and appearance contingency the indirect effect (from feedback via self-esteem) was significant and the direct effect was not, our sample direct effect actually was much stronger than (social) or equal to (appearance) the indirect effect. Most likely, this counterintuitive result is a consequence of the greater power of the test for the indirect effect in comparison to the direct and total effect (e.g., Kenny & Judd, 2014); this can, at least in some circumstances, create a greater probability of making Type I errors (Loeys, Moerkerke, & Vansteelandt, 2015). Second, in studies like ours where the relationship between the mediator (self-esteem) and the dependent variable (contingency) is not under experimental control, estimates and tests for the indirect effect are vulnerable to bias due to possible common causes for mediator and outcome (Loeys et al, 2015).…”

“…Most likely, this counterintuitive result is a consequence of the greater power of the test for the indirect effect in comparison to the direct and total effect (e.g., Kenny & Judd, 2014); this can, at least in some circumstances, create a greater probability of making Type I errors (Loeys, Moerkerke, & Vansteelandt, 2015). Second, in studies like ours where the relationship between the mediator (self-esteem) and the dependent variable (contingency) is not under experimental control, estimates and tests for the indirect effect are vulnerable to bias due to possible common causes for mediator and outcome (Loeys et al, 2015). For example, personality differences that influence both contingency and self-esteem might inadvertently be part of our indirect effect estimate.…”