A tutorial for calculating field-specific effect size distributions

Abstract

Effect sizes are useful for understanding the magnitude of study results and for planning new studies using power analysis. But despite their wide usage, effect sizes are often misinterpreted. This is mostly due to an over-reliance on general effect size benchmarks that were never intended for broad application across a range of research fields. Inaccurate effect size interpretations can consequently lead to incorrect conclusions regarding the magnitude of study results, as well as incorrect sample size estimates, which can increase the likelihood of false positive results. The aim of this article is to introduce the ESDist R package, which is designed to calculate empirically derived effect size benchmarks or a range of reliably detectable empirical effect sizes for a specific research question or field of interest. This package can be used on data that can be easily extracted from pre-existing meta-analyses to help researchers more accurately plan new studies or to better understand how an individual study might relate to other studies in their field of interest. ESDist includes a set of features that make it easy to use in a priori power analysis. Moreover, the package contains a means to estimate effect size benchmarks that account for publication bias. A companion screencast video is also provided to demonstrate the use of the ESDist package.