Simulation-Based Power Analyses for the Smallest Effect Size of Interest: A Confidence-Interval Approach for Minimum-Effect and Equivalence Testing

Abstract

Effect sizes are often used in psychology because they are crucial when determining the required sample size of a study and when interpreting the implications of a result. Recently, researchers have been encouraged to contextualize their effect sizes and determine what the smallest effect size is that yields theoretical or practical implications, also known as the “smallest effect size of interest” (SESOI). Having a SESOI will allow researchers to have more specific hypotheses, such as whether their findings are truly meaningful (i.e., minimum-effect testing) or whether no meaningful effect exists (i.e., equivalence testing). These types of hypotheses should be reflected in power analyses to accurately determine the required sample size. Through a confidence-interval-focused approach and simulations, I show how to conduct power analyses for minimum-effect and equivalence testing. Moreover, I show that conducting a power analysis for the SESOI might result in inconclusive results. This confidence-interval-focused simulation-based power analysis can be easily adopted to different types of research areas and designs. Last, I provide recommendations on how to conduct such simulation-based power analyses.

Extending Lawson and Robins’ (2021) Guideline for the Evaluation of Jingle and Jangle Fallacies: A Methodological Comment

“…The sample size determines whether an observed correlation difference yields significance at a specified level. Benchmarks for the evaluation of effect sizes neglecting the concrete context appear useless, suggesting that researchers should justify which effect size they deem non-trivial for the concrete study (Riesthuis, 2024), and subject this effect size to a priori power analyses. The R package diffcor (version 0.8.3; Blötner, 2024) provides a Monte Carlo-based power analysis function for correlation difference tests for dependent correlations (see supplemental template R file: https://osf.io/v5dte/).…”

“…Note that, depending on the intercorrelation of the ostensibly jingled/jangled constructs, this procedure requires larger sample sizes than approaches purported to ensure sufficient power to detect a bivariate correlation of pre-specified size. That is, a correlation as high as r = .10 yields significance at a smaller sample size than a correlation difference as high as Δr = .10, provided equal target power (see also Riesthuis, 2024). However, it stands to reason that robust, consequential, and far-reaching research per se requires and deserves well-powered studies.…”