Assessing fidelity of treatment delivery in group and individual 12-step facilitation

Abstract

Twelve Step Facilitation (TSF) is an emerging, empirically supported treatment, the study of which will be strengthened by rigorous fidelity assessment. This report describes the development, reliability and concurrent validity of the Twelve Step Facilitation Adherence Competence Empathy Scale (TSF ACES), a comprehensive fidelity rating scale for group and individual TSF treatment developed for the National Drug Abuse Treatment Clinical Trials Network study, Stimulant Abuser Groups to Engage in 12-Step. Independent raters used TSF ACES to rate treatment delivery fidelity of 966 (97% of total) TSF group and individual sessions. TSF ACES summary measures assessed therapist treatment adherence, competence, proscribed behaviors, empathy and overall session performance. TSF ACES showed fair to good overall reliability; weighted kappa coefficients for 59 co-rated sessions ranged from .31–1.00, with a mean of .69. Reliability ratings for session summary measures were good to excellent (.69–.91). Internal consistency for the instrument was variable (.47–.71). Relationships of the TSF ACES summary measures with each other, as well as relationships of the summary measures with a measure of therapeutic alliance provided support for concurrent and convergent validity. Implications and future directions for use of TSF ACES in clinical trials and community treatment implementation are discussed.

The relationship of therapeutic alliance and treatment delivery fidelity with treatment retention in a multisite trial of twelve-step facilitation.

“…Use of fidelity ratings of TSF sessions, using an instrument with known psychometric properties (Campbell, Manuel, et al, 2013) and independent raters who had undergone rigorous training are strengths of the current study. The inclusion of a measure of therapeutic alliance is also a strength, not only to assess its relationship with retention, but as a variable to control when examining the fidelity- retention relationships.…”

Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of a 12-step facilitation for stimulant users

“…Following the STAGE-12 trial, all TSF sessions were evaluated for adherence, competence and empathy by raters not affiliated with the STAGE-12 trial (i.e., external raters) using a ratings instrument based upon an expansion of the STAGE-12 TSF adherence scales (Campbell et al, 2013). The University of California, San Francisco and Oregon Health and Science University IRBs approved the procedures for this fidelity study.…”

“…One had a master’s degree, two were doctoral candidates, and one was a licensed clinical psychologist. External raters were recruited from local university graduate programs to conduct independent fidelity ratings of STAGE-12 sessions for the fidelity study (Campbell et al, 2013). Nine raters were employed over the course of the study; seven held masters degrees and two had doctoral degrees.…”

“…The expert rater was a doctoral level clinical psychologist with extensive experience training raters and conducting fidelity monitoring. Training procedures for therapists, supervisors and TSF expert raters are reported in Donovan et al, (2013); training and reliability procedures for external raters are described in Campbell et al (2013).…”

“…External raters used the Twelve Step Facilitation Adherence Competence Empathy Scale (TSF ACES: Campbell et al, 2013). The TSF ACES used the same adherence items and global session rating as those used in the STAGE-12 clinical trial, but included additional ratings of competence to accompany each adherence item (“How well did the counselor handle this item?”), proscribed behaviors (behaviors inconsistent with the counseling protocol), and a Global Empathy item adopted from the Motivational Interviewing Treatment Integrity (MITI) scale (Moyers et al, 2005).…”