Denouncing the use of field-specific effect size distributions to inform magnitude

Abstract

An effect size (ES) provides valuable information regarding the magnitude of effects, with the interpretation of magnitude being the most important. Interpreting ES magnitude requires combining information from the numerical ES value and the context of the research. However, many researchers adopt popular benchmarks such as those proposed by Cohen. More recently, researchers have proposed interpreting ES magnitude relative to the distribution of observed ESs in a specific field, creating unique benchmarks for declaring effects small, medium or large. However, there is no valid rationale whatsoever for this approach. This study was carried out in two parts: (1) We identified articles that proposed the use of field-specific ES distributions to interpret magnitude (primary articles); and (2) We identified articles that cited the primary articles and classified them by year and publication type. The first type consisted of methodological papers. The second type included articles that interpreted ES magnitude using the approach proposed in the primary articles. There has been a steady increase in the number of methodological and substantial articles discussing or adopting the approach of interpreting ES magnitude by considering the distribution of observed ES in that field, even though the approach is devoid of a theoretical framework. It is hoped that this research will restrict the practice of interpreting ES magnitude relative to the distribution of ES values in a field and instead encourage researchers to interpret such by considering the specific context of the study.

Power to the People: A Beginner’s Tutorial to Power Analysis using jamovi

“…Effect size distributions can be useful to calibrate your understanding of effect sizes in different areas but they are not without fault. Panzarella et al (2021) demonstrated that in the studies that cited effect size distributions, most used them to directly interpret the effect sizes they observed in their study (e.g., "in this study we found a 'large' effect, which means. .…”

“…The most difficult part of an a priori power analysis is justifying your smallest effect size of interest. Choosing an effect size to use in power analysis and interpreting effect sizes in your study requires subject matter expertise (Panzarella et al, 2021). You must decide what effect sizes you consider important or meaningful based on your understanding of the measures and designs in your area.…”

“…Second, there may be one key study you are modelling your project on. You could use their effect size to inform your power analysis, but as Panzarella et al (2021) warn, effect sizes are best interpreted in context, so question how similar your planned methods are. As a result from a single study, there will be uncertainty around their estimate, so think about the width of the confidence interval around their effect size and use a conservative estimate.…”

The Minimally Meaningful Effect Size: A Vital Component of Pre-Registrations

“…Researchers typically make no interpretations regarding magnitude or rely on procedures with questionable validity (Farmus et al, in press). These questionable approaches may include using benchmark standards for interpreting effect size magnitude (e.g., small, medium, and large interpretations, see Cohen, 1988) or interpreting magnitude by comparing the effect size in a given study to the distribution of effect sizes in a discipline or research area (Panzarella et al, 2021). These practices become troublesome when discussing the study's practical significance, as one's understanding of the impact of the effects is affected by missing, questionable, or flawed interpretations of magnitude (Panzarella et al, 2021).…”

“…These questionable approaches may include using benchmark standards for interpreting effect size magnitude (e.g., small, medium, and large interpretations, see Cohen, 1988) or interpreting magnitude by comparing the effect size in a given study to the distribution of effect sizes in a discipline or research area (Panzarella et al, 2021). These practices become troublesome when discussing the study's practical significance, as one's understanding of the impact of the effects is affected by missing, questionable, or flawed interpretations of magnitude (Panzarella et al, 2021). More precisely, Haphazardly Interpreting the Magnitude of Effect Sizes (HIMEing) can impact the interpretation of a particular finding by distorting the true practical significance of the effect.…”