Statistical Inferences Using Effect Sizes in Human Endothelial Function Research

Abstract

Introduction Magnitudes of change in endothelial function research can be articulated using effect size statistics. Effect sizes are commonly used in reference to Cohen’s seminal guidelines of small (d = 0.2), medium (d = 0.5), and large (d = 0.8). Quantitative analyses of effect size distributions across various research disciplines have revealed values differing from Cohen’s original recommendations. Here we examine effect size distributions in human endothelial function research, and the magnitude of small, medium, and large effects for macro and microvascular endothelial function. Methods Effect sizes reported as standardized mean differences were extracted from meta research available for endothelial function. A frequency distribution was constructed to sort effect sizes. The 25th, 50th, and 75th percentiles were used to derive small, medium, and large effects. Group sample sizes and publication year from primary studies were also extracted to observe any potential trends, related to these factors, in effect size reporting in endothelial function research. Results Seven hundred fifty-two effect sizes were extracted from eligible meta-analyses. We determined small (d = 0.28), medium (d = 0.69), and large (d = 1.21) effects for endothelial function that corresponded to the 25th, 50th, and 75th percentile of the data distribution. Conclusion Our data indicate that direct application of Cohen’s guidelines would underestimate the magnitude of effects in human endothelial function research. This investigation facilitates future a priori power analyses, provides a practical guiding benchmark for the contextualization of an effect when no other information is available, and further encourages the reporting of effect sizes in endothelial function research.

Redefining Effect Size Interpretations for Psychotherapy RCTs in Depression

“…However, our empirically established effect size criterion indicates that a small effect is associated with an effect size of 0.27. Prior ESD analyses have also demonstrated that using Cohen's defaults are smaller than empirically-derived effect sizes in diverse fields to a similar degree as the present analysis, such as heart rate variability (Quintana, 2017; small effect = 0.26), endothelial function (Cherubini and MacDonald, 2021; small effect = 0.28), and psychology (Szucs and Ioannidis, 2017; small effect = 0.29).…”

“…But despite the wide use of these effect size thresholds, Cohen's intention was for them to serve as a fallback option when effect size percentiles are unknown (Cohen, 2013;Glass et al, 1981;Thompson, 2009) and he subsequently expressed regret suggesting these thresholds in the first place (Funder and Ozer, 2019) . As the distribution of effect sizes vary from field-to-field, relying on Cohen's thresholds risks under-or over-estimating an effect size distribution (ESD) that represents the published literature for a given field (e. g., Quintana, 2017;Cherubini and MacDonald, 2021;Panjeh et al, 2023a;Panjeh et al, 2023b;Szucs and Ioannidis, 2017).…”

“…In terms of study planning, using Cohen's thresholds instead of a larger empirically-derived effect size threshold would lead to sample sizes that are larger than required, which can subject more people than necessary to study intervention risks and can lead to a misuse of resources that could be directed elsewhere. For example, a recent ESD analysis in the field of endothelial function reported a small effect size threshold of d = 0.28 (Cherubini and MacDonald, 2021). If one were to plan a future independent samples study using a d of 0.28 in a power analysis, a total of 404 participants would be required (assuming the input parameters of a two-tailed t-test, alpha of 0.05, power of 0.8, and a group sample size ratio of 1:1).…”

A Guide for Calculating Study-Level Statistical Power for Meta-Analyses

“…There are two broad-use cases for the metameta package. The first is the reevaluation of published meta-analysis (e.g., Cherubini & MacDonald, 2021; Quintana, 2020). This could be either for individual meta-analyses or for pooling several meta-analyses on the same topic or in the same research field.…”

A tutorial for calculating field-specific effect size distributions

“…An overview of some existing effect size benchmarks is presented in figures 1A and 1B. Figure 1A includes 6 studies that report effect size benchmarks using Cohen's d (Cherubini & MacDonald, 2021;Lovakov & Agadullina, 2021;Nordahl-Hansen et al, 2022;Plonsky & Oswald, 2014;Quintana, 2017;Szucs & Ioannidis, 2017). Figure 1B includes 7 studies that report effect size benchmarks using Pearson's r (Bosco et al, 2015;Brydges, 2019;Gignac & Szodorai, 2016;Lovakov & Agadullina, 2021;Paterson et al, 2016;Plonsky & Oswald, 2014;Schäfer & Schwarz, 2019).…”