Assessing the concurrent validity, inter‐rater reliability and test–re‐test reliability of the Australian Treatment Outcomes Profile (ATOP) in alcohol and opioid treatment populations

Abstract

Background and Aims The Australian Treatment Outcomes Profile (ATOP) is a brief instrument measuring recent substance use, risk profile and general health and wellbeing among clients attending alcohol and other drug (AoD) treatment services. This study evaluates the ATOP for concurrent validity, inter‐rater and test–re‐test reliability among alcohol and opioid treatment groups. Design For concurrent validity and inter‐rater reliability, participants completed an ATOP with a clinician and an ATOP plus standardized questionnaires (time‐line follow‐back, Opiate Treatment Index, Kessler‐10, 12‐item Short Form Survey, World Health Organization Quality of Life‐BREF, Personal Wellbeing Index) with a researcher within 3 days. For test–re‐test reliability, participants completed two ATOPs with a researcher within a 3‐day interval. Setting Outpatient AoD treatment centres in Australia. Participants For testing concurrent validity and inter‐rater reliability, 278 participants were recruited by advertisements in waiting‐rooms or clinician invitation during 2016 to 2018. A further 94 participants were recruited to examine test–re‐test reliability. Measurements Statistical tests used for concurrent validity and test–re‐test reliability were Pearson’s and Spearman’s rank order correlations for continuous variables, and Cohen’s κ for nominal variables. Inter‐rater reliability was assessed using Krippendorf’s α. Findings Most Australian Treatment Outcomes Profile items returned excellent or moderate validity and reliability. For the main substances used—alcohol, cannabis and benzodiazepines—concurrent validity, inter‐rater reliability and test–re‐test reliability all reached excellent or good agreement (0.72–0.96). Psychological health, physical health and quality of life showed fair to strong agreement with their comparator scales (0.47–0.85). Conclusions The Australian Treatment Outcomes Profile is a validated and reliable instrument for assessing recent substance use and clinical risk, health and welfare among alcohol and opioid clients in alcohol and other drug treatment settings. Its ability to reliably measure complex constructs, such as psychological and physical health, against longer scales makes it suitable for integration into routine clinical care, enabling regular monitoring of patient outcomes and safety parameters.

Opioid agonist treatment and patient outcomes during the COVID‐19 pandemic in south east Sydney, Australia

“…The pre‐COVID period captured data from 1 December 2019 to 9 April 2020; with follow‐up from 1 May to 11 September 2020. Participants included all patients who were enrolled in OAT in SESLHD in the pre‐COVID study period, were still a patient on 1 May 2020 and had a pre‐COVID Australian Treatment Outcome Profile (ATOP) [ 14 , 15 ] in the study period.…”

“…Median change in days of substance use variables was assessed using paired Wilcoxon signed‐rank tests. As group changes in mean substance use or health status over time do not always reflect changes in individual patients, we developed a threshold for statistically reliable change for ATOP scores based on the Jacobson and Truax's Reliable Change Index method [ 20 ] using standard deviations of the study sample pre‐COVID‐19 and the test–retest reliability statistics previously reported [ 15 ], enabling us to identify the proportion of patients who made statistically reliable increases in their substance use—calculated to be ≥4 days out of 28 at a 95% significance level, and the proportion of patients deteriorating on psychological, physical health and quality of life—calculated as a decrease of ≥2 points at the 80% significance level.…”

The development of an HIV-specific complexity rating scale

“…An existing rating scale comprising nine items and used in drug and alcohol settings, the Clinical Complexity Rating Scale (CCRS), 21 was adapted. This scale had a mixed binary and non-binary scoring system and was not specifically assessing functional impairment; therefore, the scoring was modified to ensure it was binary, to have consistency across variables, and the definitions were adapted to specifically address the issue of functional impairment.…”

Determining clinical cutoff scores for the Australian Treatment Outcomes Profile psychological health, physical health and quality of life questions

“…People who attend treatment for substance use disorders often experience significant co‐occurring problems in a range of areas, including their mental and physical health and quality of life [1–7]. The Australian Treatment Outcomes Profile (ATOP) [8–10] is a brief 21‐item instrument designed for use in routine clinical care in alcohol and other drug (AOD) treatment services, capturing key information about substance use, risk behaviours, and patient reported measures of quality of life and health over the preceding 4 weeks. When administered over the course of treatment, clinicians and clients can use it to collaboratively plan, review, and adapt treatment over time.…”

“…The ATOP was adapted from the British Treatment Outcome Profile [11] and a number of versions have been validated internationally including Chilean [12], Chinese [13], and Greek [14]. The validity and reliability of the ATOP has been established for use with people in treatment for problems associated with alcohol [8], heroin and other opioids [8,9], and cannabis [15]; in older clients who use substances [16]; and for use over the telephone [17]. In each of these studies, the three single‐item ATOP questions that record clients' ratings of their psychological health, physical health and quality of life (herein referred to as the PPQ items) demonstrated satisfactory concurrent validity with ‘gold standard’ measures such as Kessler‐10 (K10; 0.60–0.61), Short Form Survey 12 (SF‐12; 0.53–0.65) and the Personal Wellbeing Index (PWI; 0.73).…”