Application of Psychometric Theory to the Measurement of Voice Quality Using Rating Scales

Abstract

Rating scales are commonly used to study voice quality. However, recent research has demonstrated that perceptual measures of voice quality obtained using rating scales suffer from poor interjudge agreement and reliability, especially in the mid-range of the scale. These findings, along with those obtained using multidimensional scaling (MDS), have been interpreted to show that listeners perceive voice quality in an idiosyncratic manner. Based on psychometric theory, the present research explored an alternative explanation for the poor interlistener agreement observed in previous research. This approach suggests that poor agreement between listeners may result, in part, from measurement errors related to a variety of factors rather than true differences in the perception of voice quality. In this study, 10 listeners rated breathiness for 27 vowel stimuli using a 5-point rating scale. Each stimulus was presented to the listeners 10 times in random order. Interlistener agreement and reliability were calculated from these ratings. Agreement and reliability were observed to improve when multiple ratings of each stimulus from each listener were averaged and when standardized scores were used instead of absolute ratings. The probability of exact agreement was found to be approximately .9 when using averaged ratings and standardized scores. In contrast, the probability of exact agreement was only .4 when a single rating from each listener was used to measure agreement. These findings support the hypothesis that poor agreement reported in past research partly arises from errors in measurement rather than individual differences in the perception of voice quality.

Análise acústica de vozes infantis: contribuições do diagrama de desvio fonatório

“…We noted that 73.7% (n = 14) of the rough voices, 85% (n = 17) of the breathy voices, and 68.2% (n = 15) of the unstable voices fell outside of the area of normality in the PDD. However, most of the strained voices (71.4%, 15) were located within the area of normality in the diagram.…”

“…First, the judges were instructed that voices should be considered healthy when socially acceptable for children, produced naturally, without effort, noise, or instability, Multiple studies 1,9,14,15 have sought to determine the relationship between the acoustic characteristics of the voice signal and the perceived voice quality. It is increasingly important to establish the extent of the correlation between those two assessments, to which extent acoustic measures are capable of distinguishing normal from deviant voices, and even the discriminating power between different grades of voice disorder.…”

Voice attractiveness: Influence of stimulus duration and type

“…Additionally, although we measured between-subjects consistency (Cronbach's alpha), withinsubjects reliability of attractiveness ratings was not investigated, due to time constraints of the test. Recent evidence suggests that several repetitions are needed to obtain reliable responses on rating scales (Shrivastav, Sapienza, & Nandur, 2005). Repeated ratings are rarely done in voice attractiveness experiments, and this question would be worth being explored further.…”

A model for the prediction of breathiness in vowels

“…For such continua, a magnitude estimation task provides better perceptual data than a standard rating scale task, which may result in ordinal data (Shrivastav et al, 2005). Therefore, listeners were asked to judge the magnitude of breathiness for each stimulus using a free direct magnitude estimation task.…”