The challenge of training supervisors to use direct assessments of clinical competence in CBT consistently: a systematic review and exploratory training study

Abstract

Evaluating and enhancing supervisee competence is a key function of supervision and can be aided by the use of direct assessments of clinical competence, e.g. the Cognitive Therapy Scale – Revised (CTS-R). We aimed to review the literature regarding inter-rater reliability and training on the CTS and CTS-R to present exploratory data on training raters to use this measure. We employed a systematic review. An exploratory study evaluated the outcomes of a CTS-R supervisor training workshop (n = 34), including self-reported familiarity with and confidence in using the tool, and inter-rater consistency on three CTS-R subscales, pre- and post-training. CTS and CTS-R inter-rater reliability was variable, with evidence of rater training enhancing reliability, although the form, duration and frequency of such training is unclear. The exploratory study found that supervisors rated themselves as more familiar with and confident in using the CTS-R at the end of training compared to at the beginning. However, inter-rater reliability was poor at the beginning and end of the training. Rating competence requires supervisors to make qualitative judgements, which is inherently variable. Training raters has been shown to improve rater reliability, although this was not demonstrated in the exploratory study. Practice implications and future research priorities are identified.

Clinical psychology trainees' self versus supervisor assessments of practitioner competencies

“…Generally, the research on mental health practitioners' self-assessment highlights discrepancies in the degree of concordance between self-assessments and external raters (e.g., supervisors; Creed et al, 2016;Walfish, McAlister, O'donnell, & Lambert, 2012;Waltman, Frankel, & Williston, 2016), which is commonly used as a proxy for self- assessment accuracy (Neimeyer, Taylor, Rozensky, & Cox, 2014), and suggests that practitioners tend to both over-and under-estimate their level of competence (Loades & Myles, 2016). However, the literature is limited in the extent to which it can generalise to clinical psychology trainees (Creed et al, 2016;Loades & Armstrong, 2016). To start, research primarily involves practitioners other than psychologists, who have several years of clinical experience outside their training (e.g., nursing; Belar et al, 2001;McManus et al, 2012;Walfish et al, 2012), which makes it difficult to establish the effects of early clinical experience on psychology trainees' ability to self-assess (Esposito et al, 2015;Halonen et al, 2003).…”

Supporting our supervisors: a Summary and Discussion of the Special Issue on CBT supervision

“…Paper 9: Training supervisors to use direct assessments of clinical competence in CBT. Loades & Armstrong (2016) examine another training-related challenge: establishing inter-rater reliability amongst supervisors who assess therapist competence (i.e. applying the CTS and its revised version, the CTS-R).…”

Can we agree on the quality of clinical supervision? Inter-rater reliability of the Short–SAGE (Supervision: Adherence and Guidance Evaluation) scale

“…They then participated in three 3-hour Short-SAGE workshops, in which three randomly selected supervision sessions were analysed and discussed in order to promote a common understanding of the instrument and to reach scoring consensus. The training outline was based on the Short-SAGE manual, and the description of Loades and Armstrong (2016). In each workshop, the coders listened to a recorded supervision session and then discussed the ratings of each item until the rationale was clarified and consensus was reached.…”