Measures of fidelity of delivery of, and engagement with, complex, face‐to‐face health behaviour change interventions: A systematic review of measure quality

Abstract

PurposeUnderstanding the effectiveness of complex, face‐to‐face health behaviour change interventions requires high‐quality measures to assess fidelity of delivery and engagement. This systematic review aimed to (1) identify the types of measures used to monitor fidelity of delivery of, and engagement with, complex, face‐to‐face health behaviour change interventions and (2) describe the reporting of psychometric and implementation qualities.MethodsElectronic databases were searched, systematic reviews and reference lists were hand‐searched, and 21 experts were contacted to identify articles. Studies that quantitatively measured fidelity of delivery of, and/or engagement with, a complex, face‐to‐face health behaviour change intervention for adults were included. Data on interventions, measures, and psychometric and implementation qualities were extracted and synthesized using narrative analysis.ResultsSixty‐six studies were included: 24 measured both fidelity of delivery and engagement, 20 measured fidelity of delivery, and 22 measured engagement. Measures of fidelity of delivery included observation (n = 17; 38.6%), self‐report (n = 15; 34%), quantitatively rated qualitative interviews (n = 1; 2.3%), or multiple measures (n = 11; 25%). Measures of engagement included self‐report (n = 18; 39.1%), intervention records (n = 11; 24%), or multiple measures (n = 17; 37%). Fifty‐one studies (77%) reported at least one psychometric or implementation quality; 49 studies (74.2%) reported at least one psychometric quality, and 17 studies (25.8%) reported at least one implementation quality.ConclusionFewer than half of the reviewed studies measured both fidelity of delivery of, and engagement with complex, face‐to‐face health behaviour change interventions. More studies reported psychometric qualities than implementation qualities. Interpretation of intervention outcomes from fidelity of delivery and engagement measurements may be limited due to a lack of reporting of psychometric and implementation qualities. Statement of contribution What is already known on this subject? Evidence of fidelity and engagement is needed to understand effectiveness of complex interventionsEvidence of fidelity and engagement are rarely reportedHigh‐quality measures are needed to measure fidelity and engagement What does this study add? Evidence that indicators of quality of measures are reported in some studiesEvidence that psychometric qualities are reported more frequently than implementation qualitiesA recommendation for intervention evaluations to report indicators of quality of fidelity and engagement measures

Developing quality fidelity and engagement measures for complex health interventions

“…Fidelity of delivery is the extent to which interventions are delivered as planned (Borrelli, 2011). Consistent with previous research (Walton, Spector, Tombor, & Michie, 2017), engagement is used as an umbrella term to refer to whether a participant understands and can perform the required skills (receipt) and whether they can put plans into practice in daily life (enactment) (Borrelli, 2011). In this manuscript, we collectively refer to receipt and enactment as engagement to distinguish between provider behaviours (fidelity of delivery) and participant behaviours (engagement) (Walton et al, 2017).…”

“…Audio-recording all sessions and using multiple researchers to reliably rate a percentage for fidelity is the current gold standard (Lorencatto et al, 2013). To measure engagement, self-report, attendance records, and multiple measures have been used (Gearing et al, 2011;Hankonen et al, 2015;Rixon et al, 2016;Walton et al, 2017). There is currently no consensus regarding the gold standard method to measure engagement in face-to-face interventions (Walton et al, 2017).…”

“…To measure engagement, self-report, attendance records, and multiple measures have been used (Gearing et al, 2011;Hankonen et al, 2015;Rixon et al, 2016;Walton et al, 2017). There is currently no consensus regarding the gold standard method to measure engagement in face-to-face interventions (Walton et al, 2017). Different aspects of engagement can be measured in different ways.…”

“…This manuscript builds on the findings from an earlier review (Walton et al, 2017) and provides recommendations on how to develop measures of fidelity and engagement for complex health interventions, with consideration around psychometric and implementation qualities. This manuscript builds on the findings from an earlier review (Walton et al, 2017) and provides recommendations on how to develop measures of fidelity and engagement for complex health interventions, with consideration around psychometric and implementation qualities.…”

“…These psychometric and implementation qualities have been recommended (Gearing et al, 2011;Glasgow et al, 2005;Holmbeck & Devine, 2009;Lohr, 2002;Stufflebeam, 2000), yet reported infrequently (Walton et al, 2017). These psychometric and implementation qualities have been recommended (Gearing et al, 2011;Glasgow et al, 2005;Holmbeck & Devine, 2009;Lohr, 2002;Stufflebeam, 2000), yet reported infrequently (Walton et al, 2017).…”