A Comparison of Phone-Based and On-Site Assessment of Fidelity for Assertive Community Treatment in Indiana

Abstract

Objective This study investigated the reliability, validity, and role of rater expertise in phone-administered fidelity assessment instrument based on the Dartmouth Assertive Community Treatment Scale (DACTS). Methods An experienced rater paired with a research assistant without fidelity assessment experience or a consultant familiar with the treatment site conducted phone based assessments of 23 teams providing assertive community treatment in Indiana. Using the DACTS, consultants conducted on-site evaluations of the programs. Results The pairs of phone raters revealed high levels of consistency [intraclass correlation coefficient (ICC)=.92] and consensus (mean absolute difference of .07). Phone and on-site assessment showed strong agreement (ICC=.87) and consensus (mean absolute difference of .07) and agreed within .1 scale point, or 2% of the scoring range, for 83% of sites and within .15 scale point for 91% of sites. Results were unaffected by the expertise level of the rater. Conclusions Phone based assessment could help agencies monitor faithful implementation of evidence-based practices.

Comparison of Assertive Community Treatment Fidelity Assessment Methods: Reliability and Validity

“…ICCs were between .84 (Services subscale) and .96 (Human Resources subscale) for subscales and .91 for total DACTS, all slightly higher than a previous study (McGrew et al, 2011), with the 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 exception of the Services subscale. Mean absolute differences between phone and on-site scores also showed close consensus: .18 or less for all subscales and total DACTS.…”

“…For example, the ICC for phone total DACTS inter-rater reliability (.96) in this study exceeded the ICC found in an earlier study (McGrew et al, 2011) using ACT teams in a single state. In addition, the inter-rater reliability for both remote assessments was relatively close to the nearly perfect inter-rater reliability ICCs for onsite assessment demonstrated across 52 paired ratings in the National Evidence-Based Practices Project (ICC=.99) (McHugo et al, 2007).…”

“…Mean absolute differences between phone and on-site scores also showed close consensus: .18 or less for all subscales and total DACTS. The difference in total DACTS scores, arguably the most important score for classifying programs as ACT (McGrew et al, 2011;McHugo et al, 2007), between on-site and the remote assessments was only .11 (within 2.8% of total range) for phone and .17 (within 4.3% of total range) for expert-scored self-report, although the former was slightly higher than found in the earlier study (McGrew et al, 2011) (.07).. These differences still indicate relatively close consensus (greater than 95% accuracy), though perhaps not the level of equivalence required for the sole source for important policy distinctions, such as using phoneadministered DACTS to qualify for special funding.…”

“…Based on high DACTS inter-rater reliability for on-site assessment (ICC=.99) (McHugo et al, 1999) and previous work with phone methods, (McGrew et al, 2011;, we expected inter-rater reliability (consistency) above .9 and inter-rater consensus as indicated by mean absolute differences of less than .1 (2.5% of the scoring range). Based on promising results from the prior published work (McGrew et al, 2011), we also expected both remote methods to have excellent consistency with on-site results as indicated by ICCs of at least .80 and good consensus with on-site results as indicated by mean absolute differences of less than .2 (5% of the scoring range). Item-level results were also explored.…”

Comparing the Costs and Acceptability of Three Fidelity Assessment Methods for Assertive Community Treatment

“…McGrew, Stull, Rollins, Salyers, & Hicks, 2011). In a stepped approach, on-site assessments would be reserved for sites with "trigger" events, such as new team formation, significant staff turnover, or turnover in critical positions (e.g., team leader), and teams experiencing low fidelity scores or other implementation/quality problems.…”