For intervention programs that are applied in natural settings, randomization often is difficult or impossible to achieve. If treated individuals are compared with individuals from a nonrandomized comparison group, the inference of causality can be biased. Similar distributions in the relevant characteristics of the treatment and the comparison groups cannot be expected. To adjust between-group comparisons for preexisting differences, this article proposes a simple matching procedure. This procedure involves pairing of treatment and comparison individuals based on observable characteristics, using Euclidean distance scores. Application of the proposed Euclidean-distance matching (EuM) procedure to data from the Viennese E-Lecturing (VEL) project yields satisfying results. Possible generalizations and applications of the EuM procedure are discussed.
Background: In program evaluation interventions are usually examined for global effects but not always for differential effects. The reasoning behind the focus on global effects is that most of the concepts applied in data analyses, and most of the hypotheses being assessed, are variable-oriented. Consequently, existing differential effects will not be detected through global analyses based on a variable-oriented perspective. Accordingly, the intervention under evaluation will be wrongly considered to be ineffective. Purpose: This contribution, therefore, argues for the application of person-oriented analyses in addition to the common variable-oriented approach in order to identify the differential effects of an intervention during evaluation. Setting: Both approaches will be applied in the evaluation of a showcase program conducted in real life setting. The results of the two perspectives will be compared. Intervention: An internet supported program for the promotion of self-regulated learning in an academic context called Vienna E-Lecturing (VEL). Research Design: The evaluation of the VEL was conducted with a treatment-control-design. Data was collected at the beginning as well as at the end of the intervention. Data Collection and Analysis: Data regarding motivation as well as factual knowledge was collected. All motivational variables were assessed using previously published scales consisting of 3-4 items. Factual knowledge in statistics and methodology was tested regarding the aspects of reproduction, comprehension and the production of methodological content. Findings: The application of a variable-oriented perspective showed small effects with low practical relevance. Through the application of a person-oriented approach, and taking into account the theoretical framework of the intervention, three students groups which differed systematically in their learning motivation could be identified. For two of these groups the intervention was very successful. The discussion provides recommendations on how person-oriented approaches can be used systematically in program evaluation. Keywords: person-oriented approach; program evaluation; differential effects; learning motivation; variable-oriented approach; academic context
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.