Standardized or simple effect size: What should be reported?

Abstract

It is regarded as best practice for psychologists to report effect size when disseminating quantitative research findings. Reporting of effect size in the psychological literature is patchy -though this may be changing -and when reported it is far from clear that appropriate effect size statistics are employed. This paper considers the practice of reporting point estimates of standardized effect size and explores factors such as reliability, range restriction and differences in design that distort standardized effect size unless suitable corrections are employed. For most purposes simple (unstandardized) effect size is more robust and versatile than standardized effect size. Guidelines for deciding what effect size metric to use and how to report it are outlined. Foremost among these are: i) a preference for simple effect size over standardized effect size, and ii) the use of confidence intervals to indicate a plausible range of values the effect might take. Deciding on the appropriate effect size statistic to report always requires careful thought and should be influenced by the goals of the researcher, the context of the research and the potential needs of readers. understanding of the importance of an effect -in particular its practical importance (see Kirk, 1996), ii) comparison of effect sizes within or between studies, and iii) secondary analysis (e.g., power calculations or meta-analysis).The practice of reporting effect size is complicated, however, by the large number of different measures of effect size from which to select. There is a growing literature on what measure ought to be selected (e.g., Kirk, 1996;Olejnik & Algina, 2000;, but it would be unrealistic to expect many researchers to keep up with the full range of available effect size metrics. The aim of this paper is to consider how best to report effect size, with particular focus on the choice between standardized and simple effect size. Standardized measures of effect sizeA standardized measure of effect is one which has been scaled in terms of the variability of the sample or population from which the measure was taken. In constrast, simple effect size (Frick, 1999) is unstandardized and expressed in the original units of analysis. Rosenthal (1994) classifies standardized effect sizes into one of two main families: the r family and the d family. An important distinction between r and d is that in a two-group independent design when both are applicable, d but not r is not sensitive to the base rates (n 1 and n 2 ) of the groups

A Systematic Review of Courses, Training, and Interventions for Adoptive Parents

“…Furthermore, with more than 40% items rated as 'unclear' in our risk of bias assessment, incomplete reporting was considered a major problem in our review, which, in turn, affects the confidence in the results from the included studies. Second, guidelines for effect sizes are arbitrary and findings from studies should always be interpreted in terms of their practical and clinical significance (for discussion, see : Baguley, 2009). …”

Evaluating the Effectiveness of Correctional Education: A Meta-Analysis of Programs That Provide Education to Incarcerated Adults

“…Many studies and research syntheses have to create a common scale across disparate tests by converting scores to standard deviation units or z-scores, where a standard deviation is defined as the average deviation from the mean across test-takers on a given assessment. 1 In this case, however, all of the test scores are reported in grade equivalents or in forms that can be easily converted to grade equivalents, so we use these as our common metric, thereby avoiding the need to use standard deviation units for different tests (Baguley, 2009). 2 Grade-level equivalents have the additional benefit of being easily understood by policymakers and practitioners, because one unit is equal to a single, nine-month academic year of learning in a particular content area.…”

I See What You Are Doing: Student Views on Lecturer Use of Tablet PCs in the Engineering Mathematics Classroom

“…Although it is recognised that there are arguments against using interval scales and parametric methods with ordinal data, a reasoned assignment of an interval scale to the ordinal categories here can generate a useful raw score measure and aid interpretation and communication of results in the context of the study (Baguley, 2009;Velleman & Wilkinson, 1993). We are concerned here with making judgements on quality improvements, and a scale that gives a stronger positive emphasis to a desired goal of "very good" (rather than just "good"), and a stronger negative emphasis to "very poor" (rather than just "poor") may be useful.…”