Noise in the process: Assessing the evidential value of mediation analyses in marketing journals

Abstract

Mediation analysis plays a central role in marketing research due to its usefulness in helping to explain complex processes. Like other forms of inference, mediation analyses are susceptible to false positive results. This is particularly true when analytic decisions are based on the data, rather than a priori hypotheses. To assess the collective evidential value of mediation analyses in marketing, we used an approach first implemented by Götz and colleagues (2021) that (1) measures the relative proximity of confidence intervals to zero (RP) and (2) aggregates a related set of RP scores into a single distribution. For our analysis, we compared the RP distribution of top marketing journals (2018-20) to simulations of low power, adequate power, and null effects. We also compared the marketing journals to real-world data from Journal of Personality and Social Psychology (JPSP) (2018-20). We found that, in terms of evidential value, mediation analyses in marketing substantially deviated from simulations of adequate power and JPSP but were similar to simulations of low power and null effects. We propose study preregistration, corrections for multiple testing, and increased statistical power as solutions to increase evidence quality going forward.

Selecting a Within- or Between-Subject Design for Mediation: Validity, Causality, and Statistical Power

“…Additionally, the data generated in this simulation come from perfect normal distributions, do not contain notable outliers, and were not subject to any researcher degrees of freedom-all factors that can increase type I error rates in real data application settings (Finch, West, & MacKinnon, 1997;Zu & Yuan, 2010;Simmons, Nelson, & Simonsohn, 2011). Recent meta-analytic studies of mediation analysis show there is an over-abundance of confidence intervals that are closer to zero than would be expected from studies of high power or studies generated by a null distribution (Götz, et al 2021;Charlton et al, 2021). This suggests that multiple testing, publication bias, or other questionable research practices may be overpopulating published mediation results with falsepositives.…”

Combining Statistical and Causal Mediation Analysis

“…Finally, with respect to planning, researchers intending to conduct a mediation analysis should consider preregistering their, now well-developed, plan. Recent research on mediation analysis in psychology and marketing suggests there is potential evidence for questionable research practices (Götz, O'Boyle, Gonzalez-Mulé, Banks, & Bollmann, 2021;Charlton, Montoya, Price, & Hilgard, 2021). As previously mentioned, mediation analysis has been criticized throughout the field of psychology and beyond for it's flexible practices with causal claims.…”

“…Structural equation models may suffer from a trade off between precision and accuracy, such that the coefficients are estimated without bias, but the standard errors are larger (Ledgerwood & Shrout, 2011). This is particularly important because mediation models have been found to be generally low power Götz et al, 2021;Charlton et al, 2021), and so researchers may want to leverage latent variable modeling to increase power without increasing their sample sizes. Calls to increase latent variable modeling in mediation in social and personality psychology have largely gone unanswered (Ledgerwood & Shrout, 2011).…”

How multisensory perception promotes purchase intent in the context of clothing e-customisation

“…Relative proximity (RP) provides a degree of significance for confidence intervals (Götz et al, 2021). The valid confidence intervals do not overlap with zero (Hayes, 2013;Charlton et al, 2021). The results revealed that perceived arousal had a mediating effect between visual perception and purchase intent (t = 4.962, p < 0.001).…”