Evaluating therapist adherence in motivational interviewing by comparing performance with standardized and real patients.

Abstract

Objective The goal of measuring therapist adherence is to determine if a therapist can perform a given treatment. Yet the evaluation of therapist behaviors in most clinical trials is limited. Typically, randomized trials have few therapists and minimize therapist variability through training and supervision. Furthermore, therapist adherence is confounded with uncontrolled differences in patients across therapists. Consequently, the extent to which adherence measures capture differences in actual therapist adherence versus other sources of variance is unclear. Method We estimated intra-class correlations (ICCs) for therapist adherence in sessions with real and standardized patients (RPs and SPs), using ratings from a motivational interviewing (MI) dissemination trial (Baer et al., 2009) in which 189 therapists recorded 826 sessions with both patient types. We also examined the correlations of therapist adherence between SP and RP sessions, and the reliability of therapist level adherence scores with generalizability coefficients (GCs). Results ICC’s for therapist adherence were generally large (average ICC for SPs = 0.44, RPs = 0.40), meaning that a given therapist’s adherence scores were quite similar across their sessions. Both ICCs and GCs were larger for SP sessions as compared to RPs on global measures of MI adherence, such as Empathy and MI Spirit. Correlations between therapist adherence with real and standardized patients were moderate to large on three of five adherence measures. Conclusion Differences in therapist-level adherence ratings were substantial, and standardized patients have promise as tools to evaluate therapist behavior.

Dissemination of an evidence-based treatment for cancer patients: training is the necessary first step

“…It is rare for a CE program for mental health professionals to assess clinical skill via a simulation despite frequent use in medical education programs [38]. To our knowledge, only one training program has used confederate patients (CP) in assessing skills learned by mental health professionals [39]. In that study, ratings of therapists' use of motivational interviewing skills for addiction with CPs were similar to those with "live" patients, indicating that CPs could be a valid strategy for assessing therapist skills.…”

Adapting the Helpful Responses Questionnaire to Assess Communication Skills Involved in Delivering Contingency Management: Preliminary Psychometrics

“…In many therapies, such therapy delivery skills reflect how clinicians respond verbally to clients. Several validated skill assessment methods simulate this using standardized clinical stimuli—in the form a live standardized patient (Imel et al, 2014; Stimmel, Cohen, Fallar, & Smith, 2006), pre-recorded video clips (Baer et al, 2012; Rosengren, Hartzler, Baer, Wells, & Dunn, 2008), or written vignettes (Miller et al, 1991). A shared attribute of these methods is the opportunity they provide to measure targeted aspects of clinician verbal behavior when the clinician is given a consequence-free opportunity to rehearse communicative skills.…”

"It's hard to argue with a computer"

“…One notable exception has been the development of virtual standardized clients, which are conversational agents that simulate symptoms of various mental health conditions [27]. We believe that the relative lack of attention given to training and assessment is an unfortunate oversight, because most psychotherapy will continue to be delivered by human counselors for the foreseeable future, and at present, the quality of psychotherapy in the community is highly variable [22]. …”