Evaluating the training of clinical supervisors: a pilot study using the fidelity framework

Abstract

Theevaluation of supervisor training has featured weak measurement and lacked a coherent framework, limiting effectiveness. A literature review was first conducted to clarify the current status of supervisor workshop evaluations, related to the promising fidelity framework. This consists of five criteria: the workshop's design, the training (competence of the trainer), the delivery of the workshop, the learning of the participants (receipt), and the clinical practice outcomes (enactment). Second, we applied this framework to the training of supervisors (n = 17) in a cognitive-behavioural therapy (CBT) approach, by analysing one trainer leading two successive supervisors’ workshops. The review of the literature indicated that there were significant psychometric and conceptual deficiencies in the current evaluation of supervisor training. The data from the case-study analysis suggest that the manual-based workshop could be delivered with high fidelity by this trainer (e.g. the CBT approach to supervision received 89% approval). The fidelity framework provided a systematic, feasible and coherent rationale for the evaluation of supervisor training. Our preliminary findings indicated that the workshop was successful. To fulfil its promise as an improved way of evaluating supervisor training, the framework should be piloted with other trainers, instruments and workshops, using controlled designs.

Methods used to address fidelity of receipt in health intervention research: a citation analysis and systematic review

“…An assessment of participants’ understanding was included in 20 (60.6%) studies [25, 29, 31, 33–37, 39, 45, 47, 48, 50, 57, 61, 65, 67, 73, 75, 78] and an assessment of participants’ performance of intervention-related skills in 14 (42.4%) studies [33–36, 45, 47, 48, 51, 54, 56, 57, 65, 75, 78]. With regards to strategies to enhance receipt, 4 (12.1%) studies reported using a strategy to enhance participants’ understanding [41, 48, 56, 57], 7 (21.1%) to enhance performance of intervention-related skills [39, 41, 44, 47, 48, 56, 57].…”

“…Given the focus of the forward citation search on the BCC framework, the two most common ways of assessing receipt in the 55 studies reviewed were measurements of understanding, included in 26 (47.3%) studies [25, 29–31, 33–37, 39, 40, 45, 47–50, 57, 60–62, 65, 67, 70, 73, 75, 78], and of performance of skills, included in 16 (29.1%) studies [33–36, 45, 47, 48, 51, 54, 56, 57, 65, 70, 71, 75, 78]. …”

“…intervention components received or completed, problems areas discussed, advice given) in 9 (16.4%) studies [28, 32, 44, 60, 61, 67–70], satisfaction in 8 (14.5%) studies [27, 41, 49, 52, 55, 59, 65, 66], engagement (level of participation, involvement, enjoyment, or communication) in 8 (14.5%) studies [30, 39, 52, 55, 57, 66, 73, 76], attendance in 8 (14.5%) studies [31, 43, 56, 58, 64, 73, 74, 76], acceptability in 6 (10.9%) studies [26, 42, 48, 49, 63, 75], use of materials (e.g. website use, homework completed) in 4 (7.3%) studies [28, 46, 47, 51], behavioural change and/or maintenance in 4 (7.2%) studies [25, 54, 67, 71], receptivity or responsiveness in 3 (5.5%) studies [38, 62, 77], receipt of intervention materials in 3 (5.5%) studies [39, 59, 84], intention to implement learnings from the intervention in 2 studies [52, 60], telephone contacts during intervention delivery in 2 (3.6%) studies [48, 64], reaction to intervention or feedback on program in 2 (3.6%) studies [32, 39], self-efficacy or confidence in 2 (3.6%) studies [30, 61], exposure (e.g.…”

Supervising Cognitive-Behavioral Psychotherapy: Pressing Needs, Impressing Possibilities

“…Through direct observation and the use of standardized checklists (Bellg et al 2004). A small-scale but explicit application of this fidelity framework has suggested that this kind of systematic evaluation of supervisor training is feasible, albeit with weak instrumentation in this 'case study' (Culloty et al 2010). …”

A systematic review and reformulation of outcome evaluation in clinical supervision: Applying the fidelity framework.

“…Applying the Fidelity Framework to the Evaluation o f Outcomes From Clinical Supervision, With Illustrative Methods and StudiesWeingardt, Cucciare, Bellotti, & Lai (2009): Designed a "blended learning" system, featuring e-leaming with a self-paced, online course in CBT plus a series of live, online virtual group supervision sessions, designed to promote adherence Culloty, Milne, & Sheikh (2010)…”