The Reliability of Expert Diagnosis of Childhood Apraxia of Speech

Abstract

Purpose: The current standard for clinical diagnosis of childhood apraxia of speech (CAS) is expert clinician judgment. The psychometric properties of this standard are not well understood; however, they are important for improving clinical diagnosis. The purpose of this study is to determine the extent to which experts agree on the clinical diagnosis of CAS using two cohorts of children with mixed speech sound disorders (SSDs). Method: Speech samples of children with SSDs were obtained from previous and ongoing research from video recordings of children aged 3–8 years ( n = 36) and audio recordings of children aged 8–17 years ( n = 56). A total of 23 expert, English-speaking clinicians were recruited internationally. Three of these experts rated each speech sample to provide a description of the observed features and a diagnosis. Intrarater reliability was acceptable at 85% agreement. Results: Interrater reliability on the presence or absence of CAS among experts was poor both as a categorical diagnosis (κ = .187, 95% confidence interval [CI] [0.089, 0.286]) and on a continuous “likelihood of CAS” scale (0–100; intraclass correlation = .183, 95% CI [.037, .347]). Reliability was similar across the video-recorded and audio-only samples. There was greater agreement on other diagnoses (such as articulation disorder) than on the diagnosis of CAS, although these too did not meet the predetermined standard. Likelihood of CAS was greater in children who presented with more American Speech-Language-Hearing Association CAS consensus features. Conclusions: Different expert raters had different thresholds for applying the diagnosis of CAS. If expert clinician judgment is to be used for diagnosis of CAS or other SSDs, further standardization and calibration is needed to increase interrater reliability. Diagnosis may require operationalized checklists or reliable measures that operate along a diagnostic continuum. Supplemental Material: https://doi.org/10.23641/asha.23949105

Diagnostic Uncertainty in Childhood Motor Speech Disorders: A Review of Recent Tools and Approaches

“…For example, authors reporting treatment studies describing dysarthria secondary to cerebral palsy have either not discussed other speech diagnoses the children may have had or have excluded children with multiple speech diagnoses from the research [28,29]. In a recent study, Murray and colleagues [3] explored CAS diagnostic reliability by inviting CAS researchers to decide whether CAS was present or not in audio or video recordings of children with and without CAS. Individual raters were moderately reliable with their own judgements but not reliable with others.…”

“…[1••, 2•]) and the 2022 Childhood Apraxia of Speech Research Symposium (e.g. [3]). These papers have coincided with work describing current clinical use of diagnostic terminology in paediatric speech sound disorders and confidence in such diagnosis (e.g.…”