From randomized control trial to mixed methods: A practical framework for program evaluation based on methodological quality

Abstract

La evidencia utilizada al tomar decisiones sobre el diseño, implementación y evaluación en los programas de intervención debe ser metodológicamente sólida. Dependiendo del contexto de la intervención, se pueden aplicar diferentes metodologías. Sin embargo, el contexto de la intervención es a menudo inestable y, para adaptarse a las circunstancias cambiantes, se hace necesario modificar el plan original. El marco propuesto en este documento se basa en enfoques que pueden considerarse dos extremos de un continuo (diseños experimentales / cuasiexperimentales y estudios basados en metodología observacional). En condiciones de contexto de intervención inestable, esto permite tomar decisiones desde un enfoque de calidad metodológica en cuanto a diseño, medición y análisis. Las dimensiones estructurales, i.e., las unidades (participantes, usuarios), el tratamiento (actividades del programa), los resultados (incluidas las decisiones sobre los instrumentos a utilizar y la recopilación de datos), el entorno (contexto de implementación) y el tiempo se detallarán como parte del marco práctico. El presente estudio tiene como objetivo especificar el grado de correspondencia / complementariedad entre componentes en estas dimensiones estructurales de la evaluación de un programa desde una perspectiva de complementariedad práctica basada en la calidad metodológica. The evidence used when making decisions about the design, implementation and evaluation in intervention programs should be methodologically sound. Depending on the context of the intervention, different methodologies may apply. Nonetheless, the intervention context is often unstable and, to adapt to changing circumstances, it sometimes becomes necessary to modify the original plan. The framework proposed herein draws on approaches that can be considered two extremes of a continuum (experimental/quasi-experimental designs and studies based on observational methodology). In unstable intervention context conditions, this enables decisions from a methodological quality approach regarding design, measurement, and analysis. Structural dimensions, i.e., units (participants, users), treatment (program activities), outcomes (results, including decisions about the instruments to use and data gathering), setting (implementation context) and time will be detailed as part of the practical framework. The present study aims to specify the degree of correspondence/complementarity between components in these structural dimensions of a program evaluation from a practical complementarity perspective based on methodological quality.

Evaluating in the Real-World Educational Intervention to Improve Interference Control in Children with Autism Spectrum Disorder

“…The systematic observation developed was guided by scientific criteria, non-participative (no interrelation between the observer and the observed participants), and direct (complete level of perceptibility of the events from the recorded video) [ 36 ]. This study follows the quality guidelines for studies based on observational methodologies [ 37 ], specifically the evaluative ones [ 38 , 39 ].…”

A Training Intervention through a 360° Multisource Feedback Model

“…Based on its main characteristics, the MSF model differs from others with respect to the number of sources and raters who provide the feedback [24]. Thus, triangulating three or more different sources of evidence can allow the strengths of each individual source to compensate for weaknesses on the part of other sources, thereby contributing to a more accurate assessment than one based on any single source [25]. In terms of the quantity of raters, a multisource evaluation needs to be valid and reliable; Donnon et al [2] recommend a minimum of eight co-workers and 25 patients.…”