Editor's Notes Social Experiments in Practice: Introduction, Framing, and Context

Peck, Laura R.; Goldstein, Naomi E. S.

doi:10.1002/ev.20215

Cited by 3 publications

(5 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The criticism that social experiments cannot "look inside the black box" to determine which elements of an intervention generate its success ignores design and analysis alternatives capable of revealing which components of multi-faceted interventions lead to their success. For further elaboration of "black box" opening designs and analyses, see the recent special section on this topic, published in the American Journal of Evaluation volume 36, issue 4 (2015) and large portions of issue 152 of New Directions in Evaluation on Social Experiments in Practice (Peck, 2016b).…”

Section: Concern #11: Experiments Do Not Inform Questions Of Program ...mentioning

confidence: 99%

On the Feasibility of Extending Social Experiments to Wider Applications

Bell

Peck

2016

J Multidiscip Eval

View full text Add to dashboard Cite

Background: When deciding how to allocate limited funds for social programs, policymakers and program managers increasingly ask for evidence of effectiveness based on studies that rely on solid methodology, providing credible scientific evidence. The basic claim for the “social experiment”—that the “coin flip” of randomization creates two statistically equivalent groups that do not diverge except through an intervention’s effects—makes resulting estimates unbiased. Despite the transparency and conceptual strength of the experimental strategy for revealing the causal connection between an intervention and the outcomes of its participants, the wisdom or feasibility of conducting social experiments is often questioned on a variety of grounds. Purpose: This article defines 15 common concerns about the viability and policy reliability of social experiments, in order to assess how much these issues need constrain the use of the method in providing policy evidence. Setting: NA Intervention: NA Research Design: The research uses the authors’ experience designing and conducting dozens of social experiments to examine the basis for and soundness of each concern. It provides examples from the scholarly literature and evaluations in practice of both the problems posed and responses to each issue. Data Collection and Analysis: NA Findings: We conclude that none of the 15 concerns precludes substantially extending the use of randomized experiments as a means of evaluating the impacts of government and foundation social policies and programs.

show abstract

Section: Concern #11: Experiments Do Not Inform Questions Of Program ...mentioning

confidence: 99%

On the Feasibility of Extending Social Experiments to Wider Applications

Bell

Peck

2016

J Multidiscip Eval

View full text Add to dashboard Cite

show abstract

“…Randomized control trials (RCTs) have assumed a prominent role in social science research in the past two decades, including in education (Spybrook, 2013) and their use and applications are increasing (Peck and Goldstein, 2016). This is especially true in the United States, where federal funding of educational research programs is often contingent on building-in a summative program evaluation that assesses program impact using methods that limit potential for selection bias in results.…”

Section: Introductionmentioning

confidence: 99%

“…Social experiments are used widely to address questions of program efficacy. The Randomized Experiment eJournal counts 2000 + entries since its inception in 2007 (Peck and Goldstein, 2016). In the context of educational research in the United States, between 2002 and 2013, the Institute of Education Sciences funded more than 100 cluster randomized trials (Spybrook, 2013).…”

Section: Introductionmentioning

confidence: 99%

“…Also, results of causal impact evaluations are being summarized through compendia such as the federally funded What Works Clearinghouse (Institute of Education Sciences, n.d., https://ies.ed.gov/ncee/wwc/), where designs with random assignment can meet the coveted ‘without reservations’ evidence standard. It is safe to say, that experiments are a prioritized research method, and are here to stay (Peck and Goldstein, 2016).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement

Jaciw

Лин

Schellinger

2018

Methodological Innovations

View full text Add to dashboard Cite

Randomized trials of social programs yield internally valid estimates of causal impacts on key outcomes. While estimates of gross impact serve as useful summaries, program developers require deeper information to drive improvement efforts, especially when no impact is observed. The first main goal of this work is to present a seven-stage diagnostic method for assessing process bottlenecks in experiments. Designed for programs still in development, the troubleshooting sequence uses mixed methods to assess where in a program's logic model the process is compromised. It includes post-experimental methods that are built into the design, to account for impact variation and test where effects are intensifying or diminishing. The second main purpose of this work is to demonstrate one such method in detail. The approach tests the relationship between fidelity of program implementation and impact. First, levels of achieved fidelity in the treatment group are modeled in terms of informative baseline covariates. The model is then used to index fidelity in both conditions. Informed only by pre-randomization characteristics of individuals, the model-based fidelity scores are unbiased by endogeneity, and allow assessment of whether impacts on key outcomes vary by levels of fidelity. Results can help program developers focus improvement efforts. We illustrate the seven-step diagnostic process through a randomized trial of the Internet-Based Reading Apprenticeship Improving Science Education (iRAISE) program. Eighty-two high school science teachers and 1468 students were randomly assigned to a literacy program or control. There was no overall impact on achievement. Applying the diagnostic process revealed this was not due to a weak program contrast between conditions, or an inadequate assessment; rather, lower-than-expected impact was likely due to weaker than intended implementation.

show abstract

“…Particularly, the use of randomized controlled trials (RCTs) comes with many challenges, including: (a) ethical issues, such as the need to serve all population members at risk which prevents the use of a counterfactual in a program's evaluation; (b) timing issues, such as the urgency to act quickly and flexibly to deliver—and measure the impact of—an intervention response to an immediate risk or identified time‐sensitive programmatic need, thus, there is often not sufficient time to plan and execute a RCT or other rigorous design; (c) financial and/or technical resources, to implement the often resource‐intensive RCT; and (d) generalizability issues, such as implementing a rigorous impact evaluation that will yield results that cannot be extended beyond the experimental study context. Issues such as these have led to a call across the education intervention, human services programming, and international development literature to continue examining the suitability of alternative research designs for investigating the impact of interventions (e.g., Deterding & Solmeyer, ; Peck & Goldstein, ; Wynn, Dutta, & Nelson, ).…”

mentioning

confidence: 99%

Commentary: Contextualizing Alternatives to RCTs: Measuring the Impact of a Non‐Governmental Sponsorship Organization's Projects to Strengthen Children's Supportive Environments

Strouse¹,

Moore²

2019

New Directions for Child and Adolescent Development

View full text Add to dashboard Cite

ChildFund International (ChildFund) is a child‐focused International Non‐Governmental Organization (INGO) which, since 1938, has worked with local implementing partners (LIPs), government, and other partner organizations to help create the safe environments children need to thrive. The purpose of this commentary is to reflect on the utility and possible application of the suggestions and study designs in this special issue to real‐life intervention studies in dynamic context settings. The commentary provides three regional case examples with evaluation study lessons learned from ChildFund's global evaluation work. Specific insights from this issue are discussed regarding their application to effectiveness studies for projects delivered by a child‐focused INGO, like ChildFund, in resource‐limited settings.

show abstract

Editor's Notes Social Experiments in Practice: Introduction, Framing, and Context

Cited by 3 publications

References 15 publications

On the Feasibility of Extending Social Experiments to Wider Applications

On the Feasibility of Extending Social Experiments to Wider Applications

Diagnosing bottlenecks in development-stage field experiments: Troubleshooting and finding opportunities for improvement

Commentary: Contextualizing Alternatives to RCTs: Measuring the Impact of a Non‐Governmental Sponsorship Organization's Projects to Strengthen Children's Supportive Environments

Contact Info

Product

Resources

About