Preliminary validation of the Rating of Outcome Scale and equivalence of ultra-brief measures of well-being.

Abstract

Three brief psychotherapy outcome measures were assessed for equivalence. The Rating of Outcome Scale (ROS), a 3-item patient-reported outcome measure, was evaluated for interitem consistency, test-retest reliability, discriminant validity, repeatability, sensitivity to change, and agreement with the Outcome Rating Scale (ORS) and Outcome Questionnaire (OQ) in 1 clinical sample and 3 community samples. Clinical cutoffs, reliable change indices, and Bland-Altman repeatability coefficients were calculated. Week-to-week change on each instrument was compared via repeated-measures-corrected effect size. Community-normed T scores and Bland-Altman plots were generated to aid comparisons between instruments. The ROS showed good psychometric properties, sensitivity to change in treatment, and discrimination between outpatients and nonpatients. Agreement between the ROS and ORS was good, but neither the agreement between these nor that between ultrabrief instruments and the OQ were as good as correlations might suggest. The ROS showed incremental advantages over the ORS: improvements in concordance with the OQ, better absolute reliability, and less oversensitivity to change. The ROS had high patient acceptance and usability, and scores showed good reliability, cross-instrument validity, and responsiveness to change for the routine monitoring of clinical outcomes. (PsycINFO Database Record

Psychometric Properties of the Outcome Rating Scale (ORS) in a Spanish Clinical Sample

“…In terms of internal consistency, the results show that is strong, with a high homogeneity and cohesion of their items. The latter is in line with the findings of other studies (Anker et al, 2010;Bringhurst et al, 2006;Hafkenscheid et al, 2010;Janse et al, 2014;Miller et al, 2003;Reese, Usher, et al, 2009), including previous Spanish translation (Donoso & Grez, 2006) and the phenomena observed with other ultra-brief scales (Boulet & Boss, 1991;Seidel, Andrews, Owen, Miller, & Buccino, 2017). Cronbach's alpha coefficients are similar Table 2.…”

The effect of using the Partners for Change Outcome Management System as feedback tool in psychotherapy—A systematic review and meta-analysis

“…As mentioned in the introduction, several researchers have questioned the validity of the ORS as an outcome measure and found that the ORS yields larger ESs than other outcome measures. Seidel et al (2017) found in a small clinical sample of 73 clients that the ES from session one to three was 0.83 with the ORS and 0.44 with the OQ-45. Moreover, a distressed subsample of 28 clients from a community sample achieved an ES of 0.59 on the ORS at time 3 without receiving any treatment.…”

“…Thus, the PCOMS might create too many "signal cases," which might blur the therapist's attention, or as discussed above, give rise to demoralization. Seidel et al (2017) re-calculated the RCI taking the test-retest reliability into account and found an ORS raw score RCI of 10.7, compared to an RCI of 5 (Miller and Duncan, 2004) used in most of the included studies.…”

“…Both PCOMS's scales have acceptable psychometric properties. For the ORS, the internal consistency has been reported in at least 14 studies with Cronbach's alphas ranging from .81 (Seidel, Andrews, Owen, Miller, & Buccino, 2017) to .93 (Miller, Duncan, Brown, Sparks, & Claud, 2003). The convergent validity between the ORS total score and the OQ-45 total score has been estimated in at least four studies with correlations between .57 (Bringhurst, Watson, Miller, & Duncan, 2006) and .76 (Campbell & Hemsley, 2009).…”

“…Thus, some authors have argued that it could be challenging to use the PCOMS in the treatment of more severely disordered clients, since such clients might prefer expert advice and find it difficult to reflect on and discuss the feedback (e.g., van Oenen et al, 2016). Other researchers have questioned the validity of the ORS as an outcome measure and found that the ORS yields larger ESs than other outcome measures (Janse, Boezen-Hilberdink, van Dijk, Verbraak, & Hutschemaekers, 2014;Rise, Eriksen, Grimstad, & Steinsbekk, 2016;Seidel et al, 2017). The ORS is usually completed in the presence of the therapist, which might lead to social desirability effects due to the clients' wish to please the therapist and portray themselves or the therapist in a more positive light.…”