Validity of Clinician’s Self-Reported Practice Elements on the Monthly Treatment and Progress Summary

Abstract

An increased demand for accountability in community mental health systems has resulted in a need for valid, reliable measures of therapeutic practice. The Monthly Treatment and Progress Summary (MTPS), developed through the Hawaii Child and Adolescent Mental Health Division, is a clinician-report measure that describes therapeutic practices, treatment targets, and progress ratings for each treatment case. The current study evaluated the validity of the therapeutic strategies reported on the MTPS by comparing coder- and clinician-reported use of practices. Using 47 audio recordings from 19 youths' therapy sessions, trained observers reliably coded 12 discrete practices. Four of the 12 practices were found to be valid according to clinician-coder agreement (intraclass correlations ≥0.60). The coding system was revised, utilizing extensiveness and experiential scales, and 100 % of practices were valid according to clinician-coder agreement. Practical use of the MTPS, implications for service systems, and future directions for research on the MTPS are discussed.

Engineering Youth Service System Infrastructure: Hawaii's Continued Efforts at Large-Scale Implementation Through Knowledge Management Strategies

“…Change in therapists' mean treatment target progress ratings across treatment episodes was positively correlated with other measures of treatment progress completed by other key informants (e.g., CAFAS scores completed by care coordinators; Nakamura et al, 2007). Finally, in a just-completed study, therapists' reported practices on the MTPS were significantly associated with audiotape observers' judgments of these same practices (Borntrager, Chorpita, Orimoto, Love, & Mueller, 2013).…”

Evidence-Based Treatment Strategies in Youth Mental Health Services: Results from a National Survey of Providers

“…Prior research suggests that providers may over-report their use of treatment strategies in a given treatment session compared to trained coders (e.g., Hogue, Dauber, Lichvar, Bobek, & Henderson, 2015;Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010). Interestingly, studies looking at a longer time span (e.g., a month) suggest that while providers still report more strategy use than trained coders, they are generally consistent with treatment experts (Chapman, McCart, Letourneau, & Sheidow, 2013) and have reached acceptable reliability with trained coders (Borntrager et al, 2015). Further, while some of these previous studies asked providers about their use of prescribed strategies (i.e., strategies providers were supposed to implement), we strove to minimize social desirability bias by asking about a wide range of strategies, none of which were labeled as "evidence-based."…”

“…Evidence-Based Behavioral Strategies Scale.-While self-report measures have been developed to assess for general therapeutic approach or orientation (e.g., Therapy Procedures Checklist; TPC; Weersing, Weisz, & Donenberg, 2002) and the presence of specific strategies (Monthly Treatment Progress Summary; Borntrager, Chorpita, Orimoto, Love, & Mueller, 2015), there were no measures to assess for frequency of EBT strategy use over the course of treatment for a given youth with these MH problems. To identify EBT strategies for anxiety, depression, and disruptive behavior problems for youth 3 to 17 years old, we followed the procedures usedby Garland et al (2008) to identify EBT strategies for 4-13-year-old youth with disruptive behavior problems.…”

Characterizing Intervention Strategies Used in Community-Based Mental Health Care for Infants and Their Families

“…The current findings should be interpreted within the context of study limitations. First, the reliance on clinician self-report to estimate use of intervention strategy is one potential limitation, as previous research has found limited concordance between therapist self-rated use of strategies and observer ratings (Borntrager, Chorpita, Orimoto, Love, & Mueller, 2013;Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010). As discussed by Garland and colleagues (2010), although direct assessment of psychotherapy practice (e.g., live observation, audio-or video-recording and coding) is potentially more objective compared to indirect assessment (e.g., therapist and/or client self-report, chart/record review), it is also more costly.…”