Text as a Supplement to Speech in Young and Older Adults

Abstract

Objective The purpose of this experiment was to quantify the contribution of visual text to auditory speech recognition in background noise. Specifically, we tested the hypothesis that partially accurate visual text from an automatic speech recognizer could be used successfully to supplement speech understanding in difficult listening conditions in older adults, with normal or impaired hearing. Our working hypotheses were based on what is known regarding audiovisual speech perception in the elderly from speechreading literature. We hypothesized that: 1) combining auditory and visual text information will result in improved recognition accuracy compared to auditory or visual text information alone; 2) benefit from supplementing speech with visual text (auditory and visual enhancement) in young adults will be greater than that in older adults; and 3) individual differences in performance on perceptual measures would be associated with cognitive abilities. Design Fifteen young adults with normal hearing, fifteen older adults with normal hearing, and fifteen older adults with hearing loss participated in this study. All participants completed sentence recognition tasks in auditory-only, text-only, and combined auditory-text conditions. The auditory sentence stimuli were spectrally shaped to restore audibility for the older participants with impaired hearing. All participants also completed various cognitive measures, including measures of working memory, processing speed, verbal comprehension, perceptual and cognitive speed, processing efficiency, inhibition, and the ability to form wholes from parts. Group effects were examined for each of the perceptual and cognitive measures. Audiovisual benefit was calculated relative to performance on auditory-only and visual-text only conditions. Finally, the relationship between perceptual measures and other independent measures were examined using principal-component factor analyses, followed by regression analyses. Results Both young and older adults performed similarly on nine out of ten perceptual measures (auditory, visual, and combined measures). Combining degraded speech with partially correct text from an automatic speech recognizer improved the understanding of speech in both young and older adults, relative to both auditory- and text-only performance. In all subjects, cognition emerged as a key predictor for a general speech-text integration ability. Conclusions These results suggest that neither age nor hearing loss affected the ability of subjects to benefit from text when used to support speech, after ensuring audibility through spectral shaping. These results also suggest that the benefit obtained by supplementing auditory input with partially accurate text is modulated by cognitive ability, specifically lexical and verbal skills.

Text captioning buffers against the effects of background noise and hearing loss on memory for speech

“…Interestingly, they actually found evidence for increases in listening effort (operationalized as reaction time) with the visual task, despite subjective reports that the task was easier. In contrast, our results fit with the prior literature on assistive text captioning which suggest that visual text cues can provide a substantial benefit to speech processing (e.g., Krull & Humes, 2016;Gordon-Salant & Callahan, 2009;Grossman & Rajan, 2017).…”

“…One recent paper estimated that some popular ASR systems have word error rates between 9 and 34%; Këpuska & Bohouta, 2017). As noted previously, both Krull and Humes (2016) and Zekveld and colleagues (2008) used ASR technology to generate the text that accompanied the speech, which would allow for the introduction of realistic captioning errors.…”

The Effect of Aging and Priming on Same/Different Judgments Between Text and Partially Masked Speech

“…Captions may be helpful in a variety of conditions where, for example, the sound system is poor, there are foreign accents, there is interfering speech or noise, or the viewer’s speech perception is affected by hearing loss. Both older and younger individuals modulate their emphasis on captions versus spoken speech depending on a variety of factors, including the SNR of the acoustic signal and the accuracy of the caption (Krull & Humes, 2016). Assuming an accurate caption, one can learn the content of the spoken message simply by reading the caption text.…”

Do age and linguistic background alter the audiovisual advantage when listening to speech in the presence of energetic and informational masking?

“…In other words, spatially separating the target from the masker 'releases' the target from auditory masking. Somewhat surprisingly, providing listeners with information from a different modality prior to or during the presentation of masked sentences facilitates speech reception in noise (Freyman, Balakrishnan, & Helfer, 2004;Krull & Humes, 2016). For example, Freyman, Balakrishnan, and Helfer had participants report the last word of a syntactically correct but semantically anomalous sentence (e.g., BA rose could paint a fish.^) when it was masked by two other simultaneously presented sentences of the same type spoken in two different voices.…”