Assessing the Performance of the “Counterfactual as Self-Estimated by Program Participants”

Abstract

In this article, we test an alternative approach to creating a counterfactual basis for estimating individual and average treatment effects. Instead of using control/comparison groups or before-measures, the so-called Counterfactual as Self-Estimated by Program Participants (CSEPP) relies on program participants' self-estimations of their own counterfactual at the end of an intervention. In an empirical study, a randomized experiment was employed as a benchmark to examine the performance of the CSEPP. The results suggest that the approach is capable of delivering comparatively reliable estimates of short-and midterm effects on behavioral intentions and attitudes. With regard to the estimation of treatment effects on self-reported behavior, however, the findings are more ambiguous.

The contribution of science‐rich resources to public science interest

“…However as stated in the Methods, a number of studies from various disciplines have established that self‐report data, although not perfect are actually reasonable surrogates for more direct measures, especially when using survey data (Chan, ; Gonyea, ; Vaske, ). Similar challenges arise due to the use of retrospective data as a surrogate for actual early adolescent experiences, although as above, several investigators have found such approaches yield quite acceptable results (Lam & Bengo, ; Mueller & Gaus, ; Schwarz, ).…”

“…However as documented by a number of studies from various disciplines, self‐reports are actually reasonable surrogates for more direct measures, especially when using survey data (Chan, ; Gonyea, ; Vaske, ). The same is true for retrospective self‐reports (Lam & Bengo, ; Mueller & Gaus, ; Schwarz, ), particularly given how challenging information about past events can be to collect.…”

Evaluation at Sunset: Considerations When Evaluating a Program as It Concludes

“…This approach provides some information about possible program effects, but does not solve the second limitation, the lack of a comparison (also known as counterfactual) condition. Specifically, evaluators may not have access to individuals or groups who did not (Mueller & Gaus, 2015).…”

Full Issue

“…Many sources of existing data can support a sunset evaluation, and some can even allow for the use of a number of traditional evaluation designs (Shadish, Cook, & Campbell, 2002 (Mueller & Gaus, 2015).…”