Missed opportunities in the evaluation of public health interventions: a case study of physical activity programmes

Abstract

BackgroundEvidence-based approaches are requisite in evaluating public health programmes. Nowhere are they more necessary than physical activity interventions where evidence of effectiveness is often poor, especially within hard to reach groups. Our study reports on the quality of the evaluation of a government funded walking programme in five ‘Walking Cities’ in England. Cities were required to undertake a simple but robust evaluation using the Standard Evaluation Framework (SEF) for physical activity interventions to enable high quality, consistent evaluation. Our aim was not to evaluate the outcomes of this programme but to evaluate whether the evaluation process had been effective in generating new and reliable evidence on intervention design and what had worked in ‘real world’ circumstances.MethodsFunding applications and final reports produced by the funder and the five walking cities were obtained. These totalled 16 documents which were systematically analysed against the 52 criteria in the SEF. Data were cross checked between the documents at the bid and reporting stage with reference to the SEF guidance notes.ResultsGenerally, the SEF reporting requirements were not followed well. The rationale for the interventions was badly described, the target population was not precisely specified, and neither was the method of recruitment. Demographics of individual participants, including socio-economic status were reported poorly, despite being a key criterion for funding.ConclusionsOur study of the evaluations demonstrated a missed opportunity to confidently establish what worked and what did not work in walking programmes with particular populations. This limited the potential for evidence synthesis and to highlight innovative practice warranting further investigation. Our findings suggest a mandate for evaluability assessment. Used at the planning stage this may have ensured the development of realistic objectives and crucially may have identified innovative practice to implement and evaluate. Logic models may also have helped in the development of the intervention and its means of capturing evidence prior to implementation. It may be that research-practice partnerships between universities and practitioners could enhance this process. A lack of conceptual clarity means that replicability and scaling-up of effective interventions is difficult and the opportunity to learn from failure lost.

Study Protocol for the Evaluation of “SuperFIT”, a Multicomponent Nutrition and Physical Activity Intervention Approach for Preschools and Families

“…SuperFIT was developed in close collaboration between practice professionals and health promotion experts, in co-creation with the target group. This enhanced its applicability and usability [73], and a rigid evaluation of the program was ensured [74]. The SuperFIT approach was developed to be adaptable to the specific situation of a childcare organization and location, which may foster sustainability [75].…”

A systematic review of the use and reporting of evaluation frameworks within evaluations of physical activity interventions

“…There has been continued calls for better evaluation and reporting within public health [ 23 , 24 ]. In particular, the need for more detailed descriptions of intervention components and contextual factors to help evaluate how, why and in what contexts interventions may be effective, and to allow implementation of good practice [ 21 , 25 ].…”

“…Two previous reviews have focused specifically on the use of RE-AIM [ 26 ] and the SEF for physical activity interventions [ 23 ]. These reviews concluded that the reporting of framework components was inconsistent, and that details related to participants, recruitment and broader effects were particularly poorly reported, despite these being components of the frameworks used.…”

“…The second objective was to appraise the quality of reporting with regards to how evaluation framework use has been reported. Previous reviews [ 23 , 26 ] have assessed use of a single evaluation framework against the criteria specified in that framework. To our knowledge, no previous review has developed a set of generic indicators to facilitate the appraisal of the use of multiple evaluation frameworks in reported studies.…”

Health economic assessment of a scenario to promote bicycling as active transport in Stockholm, Sweden

“…A number of economic assessments have suggested that travel-related/transport-related interventions are cost-effective in terms of costs per healthy life-year gained in adult populations 7 18. Although, not without methodological challenges, there is a need to evaluate the costs and benefits of interventions that promote physical activity through active transport from a health sector perspective 19 20…”