Toward Policy-Relevant Benchmarks for Interpreting Effect Sizes: Combining Effects With Costs

Abstract

The common reporting of effect sizes has been an important advance in education research in recent years. However, the benchmarks used to interpret the size of these effects—as small, medium, and large—do little to inform educational administration and policy making because they do not account for program costs. The author proposes an approach to establishing cost-effectiveness benchmarks rooted in an explicit economics-based decision-making framework and assumptions about the decision-making context. To be considered large, the ratio of effects to costs must be at least as large as the ratios for substitute interventions. Evidence related to class size, prekindergarten, and other interventions is discussed to illustrate the calculation of the cost-effectiveness ratios, how the evidence can be used to develop benchmarks, and how the benchmarks can be useful for researchers and policy makers. The development of benchmarks is intended to encourage cost-effectiveness analysis as a standard part of policy analysis, thereby providing more evidence to increase the validity of the benchmarks and, ultimately, improving policy decisions. Recent cost-effectiveness research in health care policy illustrates the potential value of cost-effectiveness benchmarks in education.

Examining the Influence of a Mobile Learning Intervention on Third Grade Math Achievement

“…The costs and outcomes of the evaluand's MLI also compare favorably to those of alternative programming. This is an important criterion for judging merit and worth (Harris, 2009;Hill, Bloom, Black, & Lipsey, 2007). For example, the MLI cost per student ($252) is fiscally preferably to one-to-one laptop programs and similar initiatives that require significantly more investment in hardware, software, and technical support.…”

“…Attention should be paid to the interaction of mobile learning and several covariates, including student demographics, student math-related effort/ attitude, and teacher instructional technology background. Further cost analysis is needed to assist policyrelated decisions and provide data for cost benchmarking (Harris, 2009).…”

Cost-Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research

“…When they do make decisions based on evidence, they typically rely only on evidence of effectiveness. Tsang (1997), Levin (2001Levin ( , 2011, Levin and Belfield (2015), Levin and McEwan (2001), and Harris (2009), have argued that both costs and effects must be evaluated when considering educational interventions. By selecting programs with the highest effectiveness relative to their costs, education decision makers could improve the productivity of education, perhaps by large magnitudes.…”

A review of research on Project STAR and path ahead

“…If this research frontier can be explored, more attention must be paid to the cost-effectiveness of CSR in its best combinations relative to other school policies. Harris (2009) provided some insight, but did not consider CSR in its best combinations. STAR has greatly contributed to the understanding of CSR effects; however, it is time to move on to the topics mentioned (i.e., relations between class size and classroom processes and the pedagogical combinations best suited for different-sized classes) -thanks to the research on STAR.…”