Reflections on Statistical and Substantive Significance, with a Slice of Replication

Robinson, Daniel H.; Levin, Joel R.

doi:10.2307/1176544

Cited by 70 publications

(89 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Eff ect sizes provide information about the importance of a diff erence or relationship, and on the magnitude of results in relation to sample size and error. They have been suggested by some as more infor mative than knowledge that a statistical test is signifi cant (Carver, 1978;Chow, 1988;Levin, 1998;Robinson & Levin, 1997). Table 3 and Figure 1 present the eff ect sizes and confi dence intervals across school and home sett ings for clients who represent 0, 1, and 2 or more forms of diversity.…”

Section: Design and Data Analysismentioning

confidence: 99%

An examination of the efficacy of conjoint behavioral consultation with diverse clients.

Sheridan¹,

Eagle

Doll

2006

School Psychology Quarterly

View full text Add to dashboard Cite

This study explored the efficacy of conjoint behavioral consultation (CBC) by assessing objective outcomes and social validity with a sample of students with and without diversity. Diversity characteristics that were investigated included ethnicity, socioeconomic status, family composition, maternal education level, and language spoken in the home. Behavioral change, goal attainment, acceptability, satisfaction, and perceptions of efficacy of the CBC model were measured with 125 students representing varying levels of diversity, and 192 target behaviors. Data were collected across 8 years of a federally funded training program across two states. Findings indicated that CBC-mediated interventions yielded generally high effect sizes regardless of the presence of diversity or the number of diverse characteristics exhibited. Social validity measures also yielded very favorable results, suggesting that participants (teachers and family members, including those who experienced some form of diversity) found the procedures positive. Implications for research and practice are presented.The demographic composition of the United States is becoming increasingly diverse. Currently, 38% of elementary school children areThe research described herein was supported by federal grants awarded to the first author by the U.S. Department of Education, Office of Special Education and Rehabilitative Services. The statements and opinions contained herein belong to the authors and are not reflective of the granting agency.The authors would like to acknowledge the several parents, teachers, graduate students, and children who all contributed greatly to the completion of this study.

show abstract

Section: Design and Data Analysismentioning

confidence: 99%

An examination of the efficacy of conjoint behavioral consultation with diverse clients.

Sheridan¹,

Eagle

Doll

2006

School Psychology Quarterly

View full text Add to dashboard Cite

show abstract

“…Effect size measures as a criterion for practical significance has been recommended for a long time to supplement null hypothesis significance testing (NHST) Kirk, 1996;Plucker, 1997;Robinson & Levin, 1997;Thompson, 1998bThompson, , 1999cThompson, , 1999dThompson & Snyder, 1997). The effectiveness of this recommendation is worthy of a methodological review.…”

mentioning

confidence: 99%

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Sun¹,

Pan²,

Wang³

2010

Journal of Educational Psychology

245

146

View full text Add to dashboard Cite

Null hypothesis significance testing has dominated quantitative research in education and psychology. However, the statistical significance of a test as indicated by a p-value does not speak to the practical significance of the study. Thus, reporting effect size to supplement p-value is highly recommended by scholars, journal editors, and academic associations. As a measure of practical significance, effect size quantifies the size of mean differences or strength of associations and directly answers the research questions. Furthermore, a comparison of effect sizes across studies facilitates meta-analytic assessment of the effect size and accumulation of knowledge. In the current comprehensive review, we investigated the most recent effect size reporting and interpreting practices in 1,243 articles published in 14 academic journals from 2005 to 2007. Overall, 49% of the articles reported effect size-57% of which interpreted effect size. As an empirical study for the sake of good research methodology in education and psychology, in the present study we provide an illustrative example of reporting and interpreting effect size in a published study. Furthermore, a 7-step guideline for quantitative researchers is also summarized along with some recommended resources on how to understand and interpret effect size. Burnham, & Thompson, 2000;Kirk, 1996;Plucker, 1997;Robinson & Levin, 1997;Thompson, 1998bThompson, , 1999cThompson, , 1999dThompson & Snyder, 1997). The effectiveness of this recommendation is worthy of a methodological review. Thus, the purpose of the present study was to investigate the effect size reporting and interpreting practices in academic journals in education and psychology areas. To build the present study into a solid theoretical framework, we review the problems and misuses of NHST, and we emphasize the importance of effect size and its relationship with NHST, confidence intervals, and meta-analytical assessment of effect size. An illustrative example of reporting and interpreting effect size in a published study is also presented. To facilitate quantitative researchers' use of effect size, we recommend resources on how to understand and use effect size, and we provide a seven-step guideline for quantitative researchers to conclude the study. Keywords NHSTNHST is a traditional and popular approach to make statistical inference about research questions (Anderson et al., 2000). It is considered to be an objective, scientific procedure for knowledge accumulation (Kirk, 1996). It frames research questions in terms of two contrasting statistical hypotheses. For instance, when the purpose is to examine the effect of a treatment, the null hypothesis states that "the experimental group and the control group are not different with respect to [a specified property of interest] and that any difference found between their means is due to sampling fluctuation" (Carver, 1978, p. 381), whereas the alternative hypothesis states the opposite for a two-tailed test of mean difference as population parameter. When...

show abstract

“…Among the more vocal advocates for the reporting of effect size measures has been Thompson (1996Thompson ( , 1997Thompson ( , 1999aThompson ( , 1999b, who has been critical of the wording used in Publication Manual of the American Psychological Association, fourth edition (American Psychological Association, 1994), which encourages, rather than requires, authors to report effect size information (p. 18). While this view of mandating effect size measures is not shared by all methodologists (e.g., Frick, 1999;Levin & Robinson, 1999;Robinson & Levin, 1997), the long-anticipated report from APA Task Force on Statistical Inference recommends that researchers ''always report effect size measures for primary outcomes'' (Wilkinson et al, 1999, p. 599).…”

mentioning

confidence: 99%

Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations

Olejnik¹,

Algina

2000

Contemporary Educational Psychology

574

370

View full text Add to dashboard Cite

Although dissatisfaction with the limitations associated with tests for statistical significance has been growing for several decades, applied researchers have continued to rely almost exclusively on these indicators of effect when reporting their findings. To encourage an increased use of alternative measures of effect, the present paper discusses several measures of effect size that might be used in group comparison studies involving univariate and/or multivariate models. For the methods discussed, formulas are presented and data from an experimental study are used to demonstrate the application and interpretation of these indices. The paper concludes with some cautionary notes on the limitations associated with these measures of effect size.

show abstract

Reflections on Statistical and Substantive Significance, with a Slice of Replication

Cited by 70 publications

References 0 publications

An examination of the efficacy of conjoint behavioral consultation with diverse clients.

An examination of the efficacy of conjoint behavioral consultation with diverse clients.

A comprehensive review of effect size reporting and interpreting practices in academic journals in education and psychology.

Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations

Contact Info

Product

Resources

About