The Intrajudge Reliability of the Perceptual Rating of Cleft Palate Speech before and after Pharyngeal Flap Surgery: The Effect of Judges and Speech Samples

Abstract

Objective In this pilot study, the reliabilities of the perceptual ratings of four types of speech samples by six judges, with and without expertise in evaluating cleft palate speech, were studied. Design Pre- and postoperative tape recordings of 15 patients with cleft lip and palate who had undergone a superiorly based pharyngeal flap operation were selected. Five speech-language pathologists and one oral and maxillofacial surgeon perceptually rated the following variables on separate 100-mm visual analog scales: hypernasality, audible nasal emission, intelligibility, mis-articulations associated with velopharyngeal insufficiency, voice quality, and the presence or absence of hyponasality. These six variables were rated in four types of speech samples: reading of three sentences, repeating after the speech pathologist of three sentences, 10 sentences containing the aforementioned material, and the same 10 sentences in paired comparison. All speech samples were rerated after 3 months by the same judges. Results Judges differed largely in the range they used in their rating. Intrajudge reliability of .56 to .78 was found for ratings of hypernasality. No significant differences in intrajudge reliability were found for the ratings with the different types of speech samples. The intrajudge reliability of a judge with expertise was not necessarily higher than of a judge without this expertise. Conclusions The improvement in speech is most reliably assessed with speech samples in paired comparison. A speech-language pathologist with expertise in evaluating cleft palate speech does not guarantee a high intrajudge reliability of the rating.

Automatic, computer‐based speech assessment on edentulous patients with and without complete dentures – preliminary results

“…Even if the listener does not understand every word or syllable of a spoken sequence, the meaning can be understood by extrapolating from contextual, pragmatic and prosodic characteristics. Although there is a good correlation of intra‐individual listeners’ evaluation of speech acceptability, it shows considerable variability, which reveals the limitations of listeners’ speech assessment (14, 15).…”

“…The auditory speech evaluation by speech therapists is state of the art in analysing speech quality as an overall phonetic outcome of dental rehabilitation (3, 11–13). But the assessment of speech disorders or intelligibility is performed perceptually and therefore lacks reliability because of differences in the speech therapists’ experience (14). This is accompanied by the problem of reproducibility of the evaluation results.…”

Cleft Audit Protocol for Speech (CAPS‐A): a comprehensive training package for speech analysis

“…These two latter studies indicate the need for training of even experienced therapists in rating and transcribing cleft speech errors. Active structured training has been advocated over several training sessions with external reference taped samples, rather than depending on the internal standard of the experienced judge (McWilliams and Philips 1979, Kent et al 1999, Keuning et al 1999, Gooch et al 2001).…”

Effect of the number of presentations on listener transcriptions and reliability in the assessment of speech intelligibility in children

“…By and large, naïve listeners are considered to generate lower scores and to show more variability among themselves than experienced listeners (Pennington and Miller ), even though Keuning et al . (), assessing the intelligibility of cleft‐palate speech, found no difference in intra‐judge reliability between judges with and without expertise. Further, age, sex and familiarity with the speaker's accent seem to have no substantial effect on intelligibility scores, at least not for dysarthric speakers (Pennington and Miller ).…”

“…In other studies, the speakers have a speech–sound disorder (Hodge and Gotzke ), or the focus is on evaluation of an intervention relating to cleft palate (Keuning et al . ). The number of participants varies between 4 and 78, and there is also variation in the characteristics of listeners and the number of listeners.…”