Does it work? Using a Meta-Impact score to examine global effects in quasi-experimental intervention studies

Abstract

The evaluation of applied psychological interventions in the workplace or elsewhere is challenging. Randomisation and matching are difficult to achieve and this often results in substantial heterogeneity within intervention and control groups. As a result, traditional comparison of group means using null hypothesis significance testing may mask effects experienced by some participants. Using longitudinal studies of coaching interventions designed to provide support for dyslexic employees, this study describes and evaluates a different approach using a Meta-Impact score. We offer a conceptual rationale for our method, illustrate how this score is calculated and analysed, and show how it highlights person-specific variations in how participants react and respond to interventions. We argue that Meta-Impact is an incremental supplement to traditional variable-centric group-wise comparisons and can more accurately demonstrate in practice the extent to which an intervention worked. Such methods are needed for applied research, where personalized intervention protocols may require impact analysis for policy, legal and ethical purposes, despite modest sample sizes.

Disability coaching in a pandemic

“…These numbers were calculated to confirm the sample behaved in a similar way to previous neurodivergent coaching samples. We confirm they are in line with the medium to large effect sizes reported by Doyle and McDowall (2015, 2022).…”

“…, 2018). Previous evaluations of A2W-style coaching (Doyle and McDowall, 2015; Doyle et al. , 2022) found significant improvements in self-reports/manager reports of executive functions-related job performance (memory skills, organisational skills, time management) and stress levels.…”

“…Neurodivergent people are more vulnerable to social exclusion than their neurotypical counterparts, with disproportionate representation in unemployment (Jensen et al, 2000;Kirby et al, 2011;Lorenz et al, 2016;Palmer and Stern, 2015) and incarceration (CJJI, 2021;Snowling et al, 2000;Young et al, 2018). Previous evaluations of A2W-style coaching (Doyle and McDowall, 2015;Doyle et al, 2022) found significant improvements in self-reports/manager reports of executive functions-related job performance (memory skills, organisational skills, time management) and stress levels. The present study revisits the same analysis, adding two more dependent variables (understanding my neurodivergent strengths and concentration) comparing the coaching effect between three cohorts: pre pandemic (2019, all face-to-face), forced-remote (2020) and choice (2021, remote/face-to-face choice, but 85% chose remote).…”