No abstract

No abstract

For over thirty years now, many evaluators have recommended making explicit the underlying assumptions about how programs are expected to work-the program theory-and then using this theory to guide the evaluation. In this chapter, we provide an overview of program theory evaluation (PTE), based on our search to find what was value-added to evaluations that used this approach. We found fewer clear-cut, full-blown examples in practice than expected, but we found many interesting variations of PTE in practice and much to recommend it. And elements of PTE, whether the evaluators use the terminology or not, are being used in a wide range of areas of concern to evaluators. Based on this review, in this chapter we discuss the practice, promise, and problems of PTE. What Is Program Theory Evaluation?Because this volume is intended to demonstrate the diversity of practice, we have used a broad definition of program theory evaluation. We consider it to have two essential components, one conceptual and one empirical. PTE consists of an explicit theory or model of how the program causes the intended or observed outcomes and an evaluation that is at least partly guided by this model. This definition, though deliberately broad, does excIude some versions of evaluation that have the word theory attached to them. It does not cover all six types of theory-driven evaluation defined by Chen (1990) but only the type he refers to as intervening mechanism evaluation. It does not include evaluations that explicate the theory behind a NEW D~KECTIUNS toit EVALUATION, no. 87, Fall 2000 Q Jossey-Bass 5

The Campbell Collaboration was founded on the principle that systematic reviews on the effects of interventions will inform and help improve policy and services. Campbell offers editorial and methodological support to review authors throughout the process of producing a systematic review. A number of Campbell's editors, librarians, methodologists and external peer reviewers contribute. Plain language summaryInterventions to reduce homelessness and improve housing stability are effectiveThere are large numbers of homeless people around the world. Interventions to address homelessness seem to be effective, though better quality evidence is required. What is this review about?There are large numbers of homeless people around the world. Recent estimates are over 500,000 people in the USA, 100,000 in Australia and 30,000 in Sweden. Efforts to combat homelessness have been made on national levels as well as at local government levels.This review assesses the effectiveness of interventions combining housing and case management as a means to reduce homelessness and increase residential stability for individuals who are homeless, or at risk of becoming homeless. What is the aim of this review?This Campbell systematic review examines the effectiveness of interventions to reduce homelessness and increase residential stability for individuals who are homeless, or at risk of becoming homeless. Forty-three studies were included in the review, 37 of which are from the USA. What studies are included?Included studies were randomized controlled trials of interventions for individuals who were already, or at-risk of becoming, homeless, and which measured impact on homelessness or housing stability with follow-up of at least one year.A total of 43 studies were included. The majority of the studies (37) were conducted in the United States, with three from the United Kingdom and one each from Australia, Canada, and Denmark. 6The Campbell Collaboration | www.campbellcollaboration.org What are the main findings of this review?Included interventions perform better than the usual services at reducing homelessness or improving housing stability in all comparisons. These interventions are: These interventions seem to have similar beneficial effects, so it is unclear which of these is best with respect to reducing homelessness and increasing housing stability. What do the findings of this review mean?A range of housing programs and case management interventions appear to reduce homelessness and improve housing stability, compared to usual services.However, there is uncertainty in this finding as most the studies have risk of bias due to poor reporting, lack of blinding, or poor randomization or allocation concealment of participants. In addition to the general need for better conducted and reported studies, there are specific gaps in the research with respect to: 1) disadvantaged youth; 2) abstinence-contingent housing with case management or day treatment; 3) non-abstinence contingent housing comparing group vs independent living; 4) Hous...

Using meta-analysis, we report on an investigation of the evaluator's influence in the treatment setting on criminal recidivism outcomes. Many evaluators and users of evaluation of social interventions worry that mixing of the roles of program developer and program evaluator may bias results reported in intervention studies in a positive direction. We first review the results of prior investigations of this issue across 50 prior meta-analyses, finding 12 that tested the impact of investigator influence in the treatment setting. Eleven of these reported that effect size increased positively, sometimes substantially so, when evaluators were influential or involved in the treatment setting. We followed this with a meta-analysis of 300 randomized field trials in individually focused crime reduction, also finding intervention studies in which evaluators who were greatly influential in the treatment setting report consistently and substantially larger effect sizes than other types of evaluators. We discuss two major views Y the Fcynical_ and Fhigh fidelity_ theories Y on why this is consistently the case, and conclude with a further agenda for research.

No abstract

Scared Straight and other programs involve organized visits to prison facilities by juvenile delinquents or at-risk kids to deter them from delinquency. Despite several research studies and reviews questioning their effectiveness, they remain in use and have now been tried in at least six nations. The authors report here on the results of a systematic review of randomized experimental tests of this program. Studies that tested any program involving the organized visits of delinquents or at-risk children to penal institutions were included. Each study had to have a no-treatment control condition with at least one outcome measure of “postvisit” criminal behavior. Using extensive search methods, the authors located nine randomized trials meeting eligibility criteria. After describing the studies and appraising their methodological quality, the authors present the narrative findings from each evaluation. A meta-analysis of prevalence rates indicates that the intervention on average is more harmful to juveniles than doing nothing. The authors conclude that governments should institute rigorous programs of research to ensure that well-intentioned treatments do not cause harm to the citizens they pledge to protect.

This study (N = 48) examined the relationship between preservice secondary teachers’ subject-matter expertise in mathematics and their judgments of students’ algebra problem-solving difficulty. As predicted by the “expert blind spot” hypothesis, participants with more advanced mathematics education, regardless of their program affiliation or teaching plans, were more likely to view symbolic reasoning and mastery of equations as a necessary prerequisite for word equations and story problem solving. This view is in contrast with students’ actual performance patterns. An examination across several subject areas, including mathematics, science, and language arts, suggests a common pattern. This article considers how teachers’ developmental views may influence classroom practice and professional development, and calls into question policies that seek to streamline the licensure process of new teachers on the basis of their subject-matter expertise.