The test-retest and interrater reliability of the Promoting the Emergence of Advanced Knowledge-Direct Training assessment for use with individuals with autism and related disabilities.

Abstract

The study evaluated the test-retest and interrater reliability of the Promoting the Emergence of Advanced Knowledge-Direct Training (PEAK-DT) assessment. In the study, the reliability of PEAK-DT scores were assessed both in terms of raw scores and age-referenced scores. To evaluate test-retest reliability, 39 participants with autism or a related disability were administered the PEAK-DT assessment on 2 occasionsduring an initial assessment and a follow-up assessment after 3 weeks to 1 month. The results suggest that PEAK-DT had high test-retest reliability both for raw scores (intraclass correlation coefficient [ICC] ϭ .987, p Ͻ .001) and age-referenced scores (ICC ϭ .973, p Ͻ .001). In addition, 10 of the participants were assessed independently by 2 observers during the initial assessment period to evaluate interrater reliability. The results also suggested that PEAK-DT had strong interrater reliability ( ϭ .981, p Ͻ .001). Taken together, the results provide support for the reliability of PEAK-DT as a behavioral assessment of directly trained language skills.

The effectiveness of the peak relational training system and corresponding changes on the VB‐MAPP for young adults with autism

“…A second limitation was that stability in the pretest phase for the PEAK‐DTA assessment was not evaluated, as the assessment was only conducted on a single occasion for two of the participants. Repeated measures on the PEAK‐DTA were not obtained to avoid testing effects, and because the test–retest reliability of the assessment has been reported in prior research (Dixon et al, ). A final limitation was the homogeneity of the participants in the investigation.…”

“…The topographies of operant skills across all assessment items include socially valid intervention targets (e.g., tacting community helpers, manding for preferred items), which were guided by the Common Core State Standards in the United States (National Governors Association Center for Best Practices & Council of Chief State School Officers, 2010). Research following the publication of the PEAK-DTA has supported the construct validity of the assessment Dixon, Stanley, Belisle, & Rowsey, 2016;, and correlations have been reported between the PEAK-DTA and measures of school readiness (Dixon, Carman, et al, 2014), vocabulary (McKeel, Rowsey, , and intelligence . In addition to the PEAK-DTA, PEAK-DT contains a 184-program curriculum that directly targets skills identified in the PEAK-DTA .…”

“…A second limitation specific to the McKeel, Dixon et al study was that only the PEAK‐DTA was used as a measure of treatment outcome. Although psychometric evaluations of the PEAK‐DTA support its validity as a measure of comprehensive language development, and test–retest evaluations absent intervention suggest that scores are unlikely to increase within 1 month (Dixon et al, ), PEAK‐DT instruction is designed to train to the PEAK‐DTA. Increases in other measures of verbal operant development would provide greater confidence in the comprehensive effectiveness of PEAK‐DT as a CTM for individuals with autism.…”

Rational skepticism: A scientific review of Witts' (2018) criticisms of the PEAK relational training system

“…Again, the Artman et al () review showed 100% of reviewed JABA studies reported IOA, and only one study in the reviewed studies reported Cohen's kappa. The same criticism also stated that no clarity exists regarding whether Dixon, Stanley, Belisle, and Rowsey () utilized Cohen's kappa to interpret IOA or IRR. Indeed, the authors explicitly stated in the article, “Interrater reliability of PEAK‐DT was assessed through Cohen's kappa” (Dixon et al, , p. 37).…”

“…The same criticism also stated that no clarity exists regarding whether Dixon, Stanley, Belisle, and Rowsey () utilized Cohen's kappa to interpret IOA or IRR. Indeed, the authors explicitly stated in the article, “Interrater reliability of PEAK‐DT was assessed through Cohen's kappa” (Dixon et al, , p. 37). Whereas other statements Witts made could be debated as potentially untrue, this specific statement was explicitly false.…”