Validity of rating scale measures of voice quality

Abstract

The validity of perceptual measures of vocal quality has been neglected in studies of voice, which focus more commonly on rater reliability. Validity depends in part on reliability, because an unreliable test does not measure what it is intended to measure. However, traditional measures of rating reliability only partially represent interrater agreement, because they cannot reflect variations or patterns of agreement for specific voice samples. In this paper the likelihood that two raters would agree in their ratings of a single voice is examined, for each voice in five previously gathered data sets. Results do not support the continued assumption that traditional rating procedures produce useful indices of listeners' perceptions. Listeners agreed very poorly in the midrange of scales for breathiness and roughness, and mean ratings in the midrange of such scales did not represent the extent to which a voice possesses a quality, but served only to indicate that listeners disagreed. Techniques like analysis by synthesis or judgment of similarity avoid decomposing quality into constituent dimensions, and do not require a listener to compare an external stimulus to an unstable internal representation, thus decreasing the error in measures of quality. Modeling individual differences in perception can increase the variance accounted for in models of quality, further reducing the error in perceptual measures. Thus such techniques may provide valid alternatives to current approaches.

Treating the speech disorder in Parkinson's disease online

“…It has also been reported that perceptual rating of vocal quality, such as hoarseness, is particularly difficult and thus less reliable than expected. 17 Future research involving larger numbers of participants exhibiting a wider range of severity of the speech disturbance may achieve different outcomes.…”

Computational prosodic markers for autism

“…Auditory-perceptual methods have fundamental problems. First, the reliability and validity of auditory-perceptual methods is often lower than desirable (e.g., Kent, 1996; Kreiman & Gerratt, 1998), due to a variety of factors. For example, it is difficult to judge one aspect of speech without interference from other aspects (e.g., judging nasality in the presence of varying degrees of hoarseness); certain judgment categories are intrinsically multidimensional, thus requiring each judge to weigh subjectively and individually these dimensions; and there is a paucity of reference standards.…”

Sources of listener disagreement in voice quality assessment

“…In that study, similarities among voices were not well predicted by traditional rating scales, or indeed by any set of static phonetic or linguistic-style ''features.'' Other studies ͑e.g., Kreiman et al, 1993;Gerratt et al, 1993;Kreiman and Gerratt, 1998͒ suggest that problems with traditional voice assessment protocols may be due to factors in addition to or instead of scale validity. For example, individual listeners are reasonably self-consistent in their judgments of specific aspects of vocal quality, but across listeners more than 60% ͑and as much as 78%͒ of the variance in ratings of voices may be due to factors other than differences among voices in the quality being rated ͑Kreiman and Gerratt, 1998͒.…”